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(54) Data communication system and method, data transmission apparatus and method, data 



receiving apparatus, received-data processing method and computer program 



(57) A system allows data transfer in which the most level information. A level of priority dependent on an ap- 
appropriate packet processing is executed according to plication is specified in an RTP payload header accord - 
the performance of a terminal. A data transmission side ing to the degree of importance of scalable-encoded da- 
stores scalable-encoded data in a packet as a payload : ta ; and a level of priority is further specified in an IP 
assigns priority-level information corresponding to a lay- header. The most appropriate packet processing based 
er level of the encoded data stored in the packet to a on the performance of a terminal is allowed. Re-lrans- 
packet header, and transmits the packet. A data receiv- mission control based on the layer level of encoded data 
ing side performs processing by referring to the priority- is also allowed. 
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Description 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

[0001] The present invention relates to data commu- 
nication systems, data transmission apparatuses, data 
receiving apparatuses, data communication methods . 
data transmission methods, received-data processing 
methods, and computer programs, and more particular- 
ly, to a data communication system, a data transmission 
apparatus, a data receiving apparatus, a data commu- 
nication method, a data transmission method, a re- 
ceived-data processing method, and a computer pro- 
gram which assign a degree of priority to encoded image 
information and transmit and receive the encoded im- 
age information. 

2. Description of the Related Art 

[0002] Various types of data transfer have been, per- 
formed through various communication media, such as 
the Internet. Recently, image data, especially moving- 
image data, has been increasingly transferred through 
networks. In general, image data, especially moving-im- 
age data, is encoded (compressed) at a transmission 
side to reduce the amount of data, and is sent to a net- 
work, and received encoded signal is decoded (decom- 
pressed) at a receiving side. 

[0003] One of the most known image compression 
methods is the Moving Pictures Experts Group (MPEG) 
compression technology. These days, technical devel- 
opments have been actively performed which relate to 
a system in which an MPEG stream is stored in IP pack- 
ets conforming to the Internet Protocol (IP), transmitted 
on the Internet, and received by communication termi- 
nals., such as PCs. PDAs, and portable telephones, or 
to an image-data transfer method in such a system. 
[0004] It is necessary to assume that data transfer is 
performed with receiving terminals having different per- 
formance, in video on demand, in streaming distribution 
of live images, and in real-time communications in video 
conferences and video phones. For example, data 
transmitted from one information transmission source is 
received by a receiving terminal having a low-resolution 
display and a low-performance CPU, such as a portable 
telephone, and displayed on the display, and is also re- 
ceived by a receiving terminal having a high-resolution 
monitor and a high-performance CPU, such as a desk- 
top personal computer, and displayed. Data transmis- 
sion is performed in this way to various receiving termi- 
nals having difference processing performance. As a 
method for allowing such various receiving terminals to 
execute receiving processing and display processing 
accordingtotheirprocessing performance, a methodfor 
hierarchically encoding transfer data has been exam- 
ined, which leads to a communication system using 



scalable encoding. 

[0005] In scalable-encoded data distribution, for ex- 
ample, encoded data to be processed only by a receiv- 
ing terminal having a high-resolution display and encod- 

5 ed data to be processed in common by both of a receiv- 
ing terminal having a high-resolution display and a re- 
ceiving terminal having a low-resolution display are 
packetized in a distinguishable manner and distributed, 
and the receiving terminals select corresponding data 

10 and process it. 

[0006] As compression and decompression methods 
which allow scalable encoding, MPEG-4 and JPEG- 
2000 can be, for example, taken. MPEG-4 will include 
a fine granuality scalability technology and it is said that 

is distribution is hierarchically possible at a low bit rate to 
a high bit rate with this scalable encoding technology. 
JPEG-2000 is based on the wavelet transform, and 
makes the best use of the features of the wavelet trans- 
form to be able to perform packetization based on spa- 

20 tial resolutions or to be able to perform hierarchical 
packetization based on image quality. Motion JPEG- 
2000 (part 3) can handle moving images as well as stilt 
images, and can save scalable data In a file format. 
[0007] In conventional data distribution systems, it is 

25 necessary for a transmission side to generate data hav- 
ing different formats corresponding to the performance 
of data receiving terminals or to prepare different data 
at the transmission rates of the data receiving terminals. 
When the above-described scalable encoding is ap- 

30 plied, data distribution is made possible at the same time 
from one file to terminals having different performance. 
[000B] When data to be distributed is image data, for 
example, since real-time response is required, the user 
datagram protocol (UDP) is used in many cases for 

35 communication on the Internet. In addition, the real-time 
transport protocol (RTP) is used in the layer above the 
layer where UDP is used, and a format defined for each 
application, that is. each encoding method, is used. 
When UDP is used, since a packet is not regenerated, 

40 unlike transmission control protocol (TCP)/IP, a packet 
loss may occur due to network congestion. 
[0009] A discrete-cosine-transform-(DCT)-based 
technology has been proposed for a specific data distri- 
bution case to which scalable encoding is applied. In this 

45 technology, DCT processing is applied to distribution in- 
formation, such as image data, to generate layers with 
a high-frequency signal, and a low-frequency signal be- 
ing distinguished, packets are generated in a high-fre- 
quency layer and a low-frequency layer, and data distri- 

50 bution is executed. 

[0010] When the proposed data distribution in which 
DCT is used for the high-frequency and low-frequency 
layer process is executed, however, receiving terminals 
can execute packet processing according to levels of 

55 priority for a high-frequency signal and a low-frequency 
signal, determined, for example, by the performance of 
the terminals. There are only two levels of priority, one 
for a high-frequency signal and the other for a low-fre- 
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quency signal. In contrast, the bandwidths of network 
fluctuate in various ways. Therefore, it is difficult to say 
that only the processing based on two levels of priority 
is sufficient for handling various types of fluctuations of 
the bandwidths of networks. In addition, the DCT-based 5 
scalable technology is not sufficient for allowing termi- 
nals having different resolutions, for example, receiving 
terminals having a large resolution difference such as a 
portable telephone and a personal computer, to execute 
the most appropriate image display processing accord- 
ing to the performance of the terminals. 
[0011] Further, since encoding is executed by using 
interframe difference information in MPEG, when a 
packet loss occurs on the Internet, for example, block 
noise unique to MPEG occurs across a plurality of 
frames. The RTP format is defined for Motion JPEG in 
the IETF document, RFC 2435. The RTP format is not 
defined for JPEG-2000 video streams. Although it is bet- 
ter to apply a different process for an error and a packet 
loss to each layer with a packet loss being taken into 
account, there was no such packetizing method. 

SUMMARY OF THE INVENTION 

[0012] The present invention has been made in con- 
sideration of the foregoing problems. It is an object of 
the present invention to provide a data communication 
system, a data transmission apparatus . a data receiving 
apparatus, a data communication method, a data trans- 
mission method, a received-data processing method : 
and a computer program which, in data distribution to 
which scalable encoding is applied, allow processing 
corresponding to various types of fluctuations of the 
bandwidths of networks to be performed, and allow, if 
an error such as a packet loss occurs on a network, a 
reduction in quality of received data to be suppressed 
as much as possible. 

[001 3] The foregoing object is achieved in one aspect 
of the present invention through the provision of a data 
communication system including a data transmission 
apparatus and a data receiving apparatus. The data 
transmission apparatus includes encoding means for 
applying scalable-encoding processing by using wave- 
let transform to an image signal, and packet generation 
means for generating a packet which stores scalable- 
encoded data generated by the encoding means and for 
specifying priority-level information corresponding to a 
layer of the scalable-encoded data of the image signal, 
stored in the generated packet, as additional information 
of the generated packet. The data receiving apparatus 
includes decoding means for applying decoding 
processing to data stored in a packet to which the infor- 
mation of a level of priority equal to or largerthan a value 
specified according to the processing performance of 
the data receiving apparatus is assigned. 
[0014] The data communication system may be con- 
figured such that the data receiving apparatus further 
includes packet-priority-level-information determination 



means for determining whether a packet has the infor- 
mation of a level of priority equal to or larger than the 
value specified according to the processing perform- 
ance of the data receiving apparatus, and the decoding 
means applies the decoding processing to data stored 
in a packet selected by the packet- priority-level -infor- 
mation determination means. 

[0015] The data communication system may be con- 
figured such that the packet generation means specifies 
the priority-level information corresponding to the layer 
of the scalable-encoded data of the image signal, in an 
RTP header. 

[0016] The data communication system may be con- 
figured such that the packet generation means specifies 
the priority-level information corresponding to the layer 
of the scalable-encoded data of the image signal, in an 
IP header. 

[0017] The data communication system may be con- 
figured such that the packet generation means specifies 
the priority-level information corresponding to the layer 
of the scalable-encoded data of the image signal, in an 
RTP header and an IP header. 

[0016] The data communication system may be con- 
figured such that the packet generation means specifies 
the priority-level information corresponding to the layer 
of the scalable-encoded data of the image signal, stored 
in the generated packet, as additional information of the 
generated packet, according to a priority-level setting 
map in which the layer of the scalable-encoded data is 
associated with a level of priority. 
[0019] The data communication system may be con- 
figured such that the data transmission apparatus fur- 
ther includes a structure for receiving receiving-terminal 
information from the data receiving apparatus, and the 
packet generation means dynamically changes the pri- 
ority-level information corresponding to the layer of the 
scalable-encoded data of the image signal, stored in the 
packet, according to the receiving-terminal information. 
[0020] The data communication system may be con- 
figured such that the data transmission apparatus dy- 
namically changes the priority-level information corre- 
sponding to the layer of the scalable-encoded data of 
the image signal, stored in the packet, according to a 
network condition. 

[0021] The data communication system may be con- 
figured such that the encoding means of the data trans- 
mission apparatus applies the scalable-encoding 
processing by using the wavelet transform to the image 
signal, and generates encoded data having different 
spatial resolutions generated by the wavelet transform, 
and the packet generation means specifies priority-level 
information corresponding to layers of the spatial reso- 
lutions of the encoded data having the. different spatial 
resolutions, as additional information of the generated 
packet. 

[0022] The data communication system may be con- 
figured such that the encoding means of the data trans- 
mission apparatus generates scalable-encoded data by 
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progressive encoding processing, and the packet gen- 
eration means specifies priority-level information corre- 
sponding to a layer specified for a progressive order of 
the scalable-encoded data generated by applying the 
progressive encoding processing to the image signal s 
and stored in the generated packet, as additional infor- 
mation of the generated packet. 

[0023] The data communication system may be con- 
figured such that the encoding means of the data trans- 
mission apparatus generates scalable-encoded data by 
progressive encoding processing in which different spa- 
tial resolutions are specified in a progressive order, and 
the packet generation means specifies priority-level in- 
formation corresponding to a layer specified for each 
spatial resolution of the scalable-encoded data of the 
image signal, stored in the generated packet, as addi- 
tional information of the generated packet. 
[0024] The data communication system may be con- 
figured such that the encoding means of the data trans- 
mission apparatus generates scalable-encoded data by 
progressive encoding processing in which different 
SNRs are specified in a progressive order, and the pack- 
et generation means specifies priority-level information 
corresponding to a layer specified for each SNR of the 
scalable-encoded data of the image signal, stored in the 
generated packet, as additional information of the gen- 
erated packet. 

[0025] The data communication system may be con- 
figured such that the encoding means of the data trans- 
mission apparatus generates scalable-encoded data by 
progressive encoding processing in which different 
color components are specified in a progressive order 
and the packet generation means specifies priority-level 
information corresponding to a layer specified for each 
colorcomponent of the scalable-encoded data of the im- 
age signal, stored in the generated packet, as additional 
information of the generated packet. 
[0026] The data communication system may be con- 
figured such that the data transmission apparatus fur- 
ther includes a re-transmission control structure for, in 
response to a packet re-transmission request sent from 
the data receiving apparatus, determining the levels of 
priority assigned to packets for which the re-transmis- 
sion request has been sent; for determining whether re- 
transmission is to be performed according to the deter- 
mined levels of priority, and for re-transmitting packets 
having higher levels of priority earlier. 
[0027] The foregoing object is achieved in another as- 
pect of the present invention through the provision of a 
data transmission apparatus for storing image data in a 
packet and for transmitting the packet, including encod- 
ing means for applying scalable-encoding processing 
by using wavelet transform to an imago signal, and 
packet generation means, for generating a packet which 
stores scalable-encoded data generated by the encod- 
ing means and for specifying priority-level information 
corresponding to a layer of the scalable-encoded data 
of the image signal, stored in the generated packet as 



additional information of the generated packet. 
[0028] The data transmission apparatus may be con- 
figured such that the packet generation means specifies 
the priority-level information corresponding to the layer 
of the scalable-encoded data of the image signal, in an 
RTP header. 

[0029] The data transmission apparatus may be con- 
figured such that the packet generation means specifies 
the priority-level information corresponding to the layer 
of the scalable-encoded data of the image signal, in an 
IP header. 

[0030] The data transmission apparatus may be con- 
figured such that the packet generation means specifies 
the priority-level information corresponding to the layer 
of the scalable-encoded data of the image signal, in an 
RTP header and an !P header. 

[0031 ] The data transmission apparatus may be con- 
figured such that the packet generation means specifies 
the priority-level information corresponding to the layer 
of the scalable-encoded data of the image signal, stored 
in the generated packet, as additional information of the 
generated packet, according to a priority-level setting 
map in which the layer of the scalable-encoded data is 
associated with a level of priority. 
[0032] The data transmission apparatus may be con- 
figured such that the data transmission apparatus fur- 
ther comprises a structure for receiving receiving-termi- 
nal information from the data receiving apparatus, and 
the packet generation means dynamically changes the 
priority- level information corresponding to the layer of 
the scalable-encoded data of the image signal, stored 
in the packet, according to the receiving-terminal infor- 
mation. 

[0033] The data transmission apparatus may dynam- 
ically change the priority- level information correspond- 
ing to the layer of the scalable-encoded data of the im- 
age signal, stored in the packet, according to a network 
condition. 

[0034] The data transmission apparatus may be con- 
figured such that the encoding means applies the scal- 
able-encoding processing by using the wavelet trans- 
form to the image signal, and generates encoded data 
having different spatial resolutions generated by the 
wavelet transform, and the packet generation means 
specifies priority-level information corresponding to lay- 
ers of the spatial resolutions of the encoded data having 
the different spatial resolutions, as additional informa- 
tion of the generated packet. 

[0035] The data transmission apparatus may be con- 
figured such that the encoding means generates scala- 
ble-encoded data by progressive encoding processing, 
and the packet generation means specifies priority-level 
information corresponding to a layer specified for a pro- 
gressive order of the scalable-encoded data generated 
by applying the progressive encoding processing to the 
image signal and stored in the generated packet, as ad- 
ditional information of the generated packet. 
[0036] The data transmission apparatus may be con- 
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figured such that the encoding means generates scala- 
ble-encoded data by progressive encoding processing 
in which different spatial resolutions are specified in a 
progressive order, and the packet generation means 
specifies priority- 1 eve I information corresponding to a 
layer specified for each spatial resolution of the scala- 
ble-encoded data of the image signal, stored in the gen- 
erated packet, as additional information of the generat- 
ed packet. 

[0037] The data transmission apparatus may be con- 
figured such that the encoding means. generates scala- 
ble-encoded data by progressive encoding processing 
in which different SNRs are specified in a progressive 
order, and the packet generation means specifies prior- 
ity-level information corresponding to a layer specified 
for each SNR of the scalable-encoded data of the image 
signal, stored in the generated packet, as additional in- 
formation of the generated packet. 
[0038] The data transmission apparatus may be con- 
figured such that the encoding means generates scala- 
ble-encoded data by progressive encoding processing 
in which different color components are specified in a 
progressive order, and the packet generation means 
specifies priority- level information corresponding to a 
layer specified for each color component of the scalable- 
encoded data of the image signal, stored in the gener- 
ated packet, as additional information of the generated 
packet. 

[0039] The data transmission apparatus may further 
include a re-transmission control structure for. in re- 
sponse to a packet re-transmission request sent from 
the data receiving apparatus, determining the levels of 
priority assigned to packets for which the re-transmis- 
sion request has been sent; for determining whether re- 
transmission is to be performed according to the deter- 
mined levels of priority, and for re-transmitting packets 
having higher levels of priority earlier. 
[0040] The foregoing object is achieved in yet another 
aspect of the present invention through the provision of 
a data receiving apparatus for receiving packets which 
store encoded data : including receiving means for re- 
ceiving packets which store data scalable-encoded by 
wavelet transform; packet-priority-level-information de- 
termination means for determining whether the packets 
have the' information of a level of priority equal to or larg- 
er than a value specified according to the processing 
performance of the data receiving apparatus; and de- 
coding means for decoding data stored in packets se- 
lected by the packet-priority-level-information determi- 
nation means. 

[0041] The data receiving apparatus may be config- 
ured such that the packet-priority-level-information de- 
termination means includes IP-packet filtering means 
for determining a level of priority assigned to the IP 
header of a received packet: and RTP-packet-header 
scanning means for determining a level of priority as- 
signed to the RTP header of the received packet. 
[0042] The foregoing object is achieved in stili another 



aspect of the present invention through the provision of 
a data communication method for a data communication 
system comprising a data transmission apparatus and 
a data receiving apparatus, including the steps of an en- 

5 coding step of applying scalable-encoding processing 
by using wavelet transform to an image signal: a packet 
generation step of generating a packet which stores 
scalable-encoded data generated in the encoding step, 
and of specifying priority-level information correspond- 

10 ing to a layer of the scalable-encoded data of the image 
signal, stored in the generated packet, as additional in- 
formation of the generated packet, both steps being ex- 
ecuted in the data transmission apparatus; and a de- 
coding step of decoding data stored in a packet to which 

15 the information of a level of priority equal to or larger 
than a value specified according to the processing per- 
formance of the data receiving apparatus is assigned, 
the decoding step being executed in the data receiving 
apparatus. 

20 [0043] The data communication method may be con- 
figured such that it further includes a packet-priority-lev- 
el-information determination step of determining wheth- 
er a packet has the information of a level of priority eq ual 
to or larger than the value specified according to the 

25 processing performance of the data receiving appara- 
tus: the packet-priority-level-information determination 
step being executed by the data receiving apparatus, 
and the decoding step decodes data stored in a packet 
selected by the packet-priority-level-information deter- 

30 mination step. 

[0044] The data communication methou ma/ be con- 
figured such that the packet generation step includes a 
step of specifying the priority-level information corre- 
sponding to the layer of the scalable-encoded data of 

35 the image signal, in an RTP header. 

[0045] The data communication method may be con- 
figured such that the packet generation step includes a 
step of specifying the priority-level information corre- 
sponding to the layer of the scalable-encoded data of 

40 the image signal, in an IP header. 

[0046] The data communication method may be con- 
figured such that the packet generation step includes a 
step of specifying the priority-level information corre- 
sponding to the layer of the scalable-encoded data of 

45 the image signal, in an RTP header and an IP header. 
[0047] The data communication method may be con- 
figured such that the packet generation step includes a 
step of specifying the priority-level information corre- 
sponding to the layer of the scalable-encoded data of 

so the image signal, stored in the generated packet, as ad- 
ditional information of the generated packet, according 
to a priority-level setting map in which the layer of the 
scalable-encoded data is associated with a level of pri- 
ority. 

55 [0048] The data communication method may be con- 
figured such that it further includes a step of receiving 
receiving-terminal information from the data receiving 
apparatus, the step being executed by the data trans- 
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mission apparatus, and the packet generation step in- 
cludes a step of dynamically changing the priority- level 
information corresponding to the layer of the scalable- 
encoded data of the image signal, stored in the packet, 
according to the receiving-terminal information. 
[0049] The data communication method may be con- 
figured such that it further includes a step of dynamically 
changing the priority-level information corresponding to 
the layer of the scalable-encoded. data of the image sig- 
nal, stored in the packet, according to a network condi- 
tion, the step being executed by the data transmission 
apparatus. 

[0050] The data communication method may be con- 
figured such that the encoding step executed in the data 
transmission apparatus includes a step of applying the 
scalable-encoding processing by using the wavelet 
transform to the image signal and of generating encod- 
ed data having different spatial resolutions generated 
by the wavelet transform, and the packet generation 
step includes a step of specifying priority-level informa- 
tion corresponding to layers of the spatial resolutions of 
the encoded data having the different spatial resolu- 
tions, as additional information of the generated packet. 
[0051] The data communication method may be con- 
figured such that the encoding step executed in the data 
transmission apparatus includes a step of generating 
scalable-encoded data by progressive encoding 
processing, and the packet generation step includes a 
step of specifying priority-level information correspond- 
ing to a layer specified for a progressive order of the 
scalable-encoded data generated by applying the pro- 
gressive encoding processing to the image signal and 
stored in the generated packet, as additional information 
of the generated packet. 

[0052] The data communication method may be con- 
figured such that the encoding step executed in the data 
transmission apparatus includes a step of generating 
scalable-encoded data by progressive encoding 
processing in which different spatial resolutions are 
specified in a progressive order and the packet gener- 
ation step includes a step of specifying priority-level in- 
formation corresponding to a layer specified for each 
spatial resolution of the scalable-encoded data of the 
image signal, stored in the generated packet, as addi- 
tional information of the generated packet. 
[0053] The data communication method may be con- 
figured such that the encoding step executed in the data 
transmission apparatus includes a step of generating 
scalable-encoded data by progressive encoding 
processing in which different SNRs are specified in a 
progressive order and the packet generation step in- 
cludes a step of specifying priority-level information cor- 
responding to a layer specified for each SNR of the scal- 
able-encoded data of the image signal, stored in the 
generated packet, as additional information of the gen- 
erated packet. 

[0054] The data communication method may be con- 
figured such thatthe encoding step executed in the data 



transmission apparatus includes a step of generating 
scalable-encoded data by progressive encoding 
processing in which different color components are 
specified in a progressive order, and the packet gener- 

5 ation step includes a step of specifying priority- level in- 
formation corresponding to a layer specified for each 
color component of the scalable-encoded data of the im- 
age signal, stored in the generated packet, as additional 
information of the generated packet. 

10 [0055] The data communication method may further 
include a step of, in response to a packet re-transmis- 
sion request sent from the data receiving apparatus, de- 
termining the levels of priority assigned to packets for 
which the re-transmission request has been sent; of de- 

15 termining whether re -transmission is to be performed 
according to the determined levels of priority, and of re- 
transmitting packets having higher levels of priority ear- 
lier, the step being executed by the data transmission 
apparatus. 

20 [0056] The foregoing object is achieved in still yet an- 
other aspect of the present invention through the provi- 
sion of a data transmission method for storing image da- 
ta in a packet and for transmitting the packet, including 
the steps of an encoding step of applying scalable-en- 

25 coding processing by using wavelet transform to an im- 
age signal, and a packet generation step, of generating 
a packet which stores scalable-encoded data generated 
in the encoding step and of specifying priority- 1 eve I in- 
formation corresponding to a layer of the scalable-en- 

30 coded data of the image signal, stored in the generated 
packet, as additional information of the generated pack- 
et. 

[0057] The data transmission method may be config- 
ured such that the packet generation step includes a 

35 step of specifying the priority- 1 eve I information corre- 
sponding to the layer of the scalable-encoded data of 
the image signal, in an RTP header. 
[0058] The data transmission method may be config- 
ured such that the packet generation step includes a 

40 step of specifying the priority- 1 eve I information corre- 
sponding to the layer of the scalable-encoded data of 
the image signal, in an IP header. 
[0059] The data transmission method may be config- 
ured such that the packet generation step includes a 

45 step of specifying the priority- 1 eve I information corre- 
sponding to the layer of the scalable-encoded data of 
the image signal, in an RTP header and an IP header. 
[0060] The data transmission method may be config- 
ured such that the packet generation step includes a 

so step of specifying the priority-level information corre- 
sponding to the layer of the scalable-encoded data of 
the image signal, stored in the generated packet, as ad- 
ditional information of the generated packet, according 
to a priority-level setting map in which the layer of the 

55 scalable-encoded data is associated with a level of pri- 
ority. 

[0061 ] The data transmission method may be config- 
ured such that it further comprises a step of receiving 
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receiving-terminal information from the data receiving 
apparatus, the step being executed by the data trans- 
mission apparatus, and the packet generation step in- 
cludes a step of dynamically changing the priority-level 
information corresponding to the layer of the scalable- 
encoded data of the image signal, stored in the packet, 
according to the receiving-terminal information. 
[0062] The data transmission method may be config- 
ured such that it further includes a step of dynamically 
changing the priority-level information corresponding to 
the layer of the scalable-encoded data of the image sig- 
nal, stored in the packet, according to a network condi- 
tion, the step being executed by the data transmission 
apparatus. 

[0063] The data transmission method may be config- 
ured such that the encoding step includes a step of ap- 
plying the scalable-encoding processing by using the 
wavelet transform to the image signal, and of generating 
encoded data having different spatial resolutions gen- 
erated by the wavelet transform, and the packet gener- 
ation step includes a step of specifying priority-level in- 
formation corresponding to layers of the spatial resolu- 
tions of the encoded data having the different spatial 
resolutions, as additional information of the generated 
packet. 

[0064] The data transmission method may be config- 
ured such that the encoding step includes a step of gen- 
erating scalable-encoded data by progressive encoding 
processing, and the packet generation step includes a 
step of specifying priority-level information correspond- 
ing to a layer specified for a progressive order of the 
scalable-encoded data generated by applying the pro- 
gressive encoding processing to the image signal and 
stored in the generated packet, as additional information 
of the generated packet. 

[0065] The data transmission method may be config- 
ured such that the encoding step includes a step of gen- 
erating scalable-encoded data by progressive encoding 
processing in which different spatial resolutions are 
specified in a progressive order and the packet gener- 
ation step includes a step of specifying priority-level in- 
formation corresponding to a layer specified for each 
spatial resolution of the scalable-encoded data of the 
image signal, stored in the generated packet, as addi- 
tional information of the generated packet. 
[0066] The data transmission method may be config- 
ured such that the encoding step includes a step of gen- 
erating scalable-encoded data by progressive encoding 
processing in which different SNRs are specified in a 
progressive order, and the packet generation step in- 
cludes a step of specifying priority-level information cor- 
responding to a layer specified for each SNR of the scal- 
able-encoded data of the image signal, stored in the 
generated packet, as additional information of the gen- 
erated packet. 

[0067] The data transmission method may be config- 
ured such that the encoding step includes a step of gen- 
erating scalable-encoded data by progressive encoding 



processing in which different color components are 
specified in a progressive order, and the packet gener- 
ation step includes a step of specifying priority- level in- 
formation corresponding to a layer specified for each 
5 color component of the scalable-encoded data of the im- 
age signal, stored in the generated packet, as additional 
information of the generated packet. 
[0068] The data transmission method may further in- 
clude a step of, in response to a packet re-transmission 
10 request sent from the data receiving apparatus, deter- 
mining the levels of priority assigned to packets for 
which the re-transmission request has been sent; of de- 
termining whether re-transmission is to be performed 
according to the determined levels of priority, and of re- 
's transmitting packets having higher levels of priority ear- 
lier, the step being executed by the data transmission 
apparatus. 

[0069] The foregoing object is achieved in a further 
aspect of the present invention through the provision of 

20 a received-data processing method for receiving pack- 
ets which store encoded data and for executing data 
processing, including the steps of a receiving step of re- 
ceiving packets which store data scalable-encoded by 
wavelet transform; a packet-priority-level-information 

25 determination step of determining whether the packets 
have the information of a level of priority equal to or larg- 
er than a value specified according to the processing 
performance of the data receiving apparatus; and a de- 
coding step of decoding data stored in packets selected 

30 in the packet-priority-level-information determination 
step. 

[0070] The received-data processing method may be 
configured such that the packet-priority-level-informa- 
tion determination step includes an IP-packet filtering 

35 step of determining a level of priority assigned to the I P 
header of each received packet, and an RTP-packet- 
header scanning step of determining a level of priority 
assigned to the RTP header of the received packet. 
[0071] The foregoing object is achieved in a yet fur- 

40 ther aspect of the present invention through the provi- 
sion of a computer program for executing processing for 
storing image data in a packet and for sending the pack- 
et, including the steps of an encoding step of. applying 
scalable-encoding processing by using wavelet trans- 

45 form to an image signal, and a packet generation step 
of generating a packet which stores scalable-encoded 
data generated in the encoding step and of specifying 
priority-level information corresponding to a layer of the 
scalable-encoded data of the image signal : stored in the 

5 o generated packet, as additional information of the gen- 
erated packet. 

[0072] A computer program according to the present 
invention can be provided, for example, for general-pur- 
pose computer systems which can execute various pro- 
55 gram codes, by storage media, communication media, 
such as networks, and recording media, such as CDs : 
FDs, and MOs. in a computer-readable form. When the 
program is provided in a computer-readable form, the 
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computer syslem executes processing according to the 
program. 

[0073] Other objects, features, and advantages of the 
present invention will be made clear by more detailed 
descriptions based on embodiments and attached 
drawings. A system in the present specification refers 
to a logical combination of a plurality of apparatuses. 
The plurality of apparatuses are not necessarily dis- 
posed in the same cabinet. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0074] 

Fig. 1 is a view showing an example network struc- 
ture to which a system according to the present in- 
vention is applied. 

Fig. 2 is a view showing an example packet distri- 
bution based on levels of priority in a network struc- 
ture to which a system according to the present in- 
vention is applied. 

Fig. 3 is a view showing an example packet distri- 
bution based on levels of priority through a satellite, 
to which a system according to the present inven- 
tion is applied. 

Fig. 4 is a block diagram of a data transmission ap- 
paratus according to the present invention. 
Fig. 5 is a view showing an example encoding- 
processing structure which uses wavelet transform. 
Fig. 6 is a view used for describing wavelet trans- 
form processing. 

Fig. 7 is a view used for describing wavelet trans- 
form processing. 

Fig. 8 is a view of the structure of data output from 
an encoder of a data transmission apparatus ac- 
cording to the present invention. 
Fig. 9 is a view showing example processing in a 
packetizer of the data transmission apparatus ac- 
cording to the present invention. 
Fig. 10 is a view showing packet generation 
processing with spatial resolutions being specified 
in progressive layers, taken as an example 
processing in the packetizer of the data transmis- 
sion apparatus according to the present invention. 
Fig. 11 is. a view showing packet generation 
processing with image quality (SNRs) being speci- 
fied in progressive layers, taken as an example 
processing in the packetizer of the data transmis- 
sion apparatus according to the present invention. 
Fig. 12 is a view showing packet generation 
processing with color components being specified 
in progressive layers, taken as an example 
processing in the packetizer of the data transmis- 
sion apparatus according to the present invention. 
Fig. 13 is a view showing the structure of the RTP 
header of a generated packet in the packetizer of 
the data transmission apparatus according to the 
present invention. 



Fig. 14 is a view showing the structure of the IP 
header of a generated packet in the packetizer of 
the data transmission apparatus according to the 
present invention. 

5 Fig. 1 5 is a view of a priority-level setting map used 

for specifying levels of priority for generated pack- 
ets in the packetizer of the data transmission appa- 
ratus according to the present invention. 
Fig. 1 6 is a view of another priority-level setting map 

10 used for specifying levels of priority for generated 
packets in the packetizer of the data transmission 
apparatus according to the present invention. 
Fig. 1 7 is a block diagram of a data receiving appa- 
ratus according to the present invention. 

15 Fig. 1 8 is a view used for describing inverse wavelet 
transform processing. 

Fig. 1 9 is a view used for describing packet re-trans- 
mission control executed between the data trans- 
mission apparatus and the data receiving appara- 
20 tus. 

Fig. 20 is a view of a processing sequence executed 
between the data transmission apparatus and the 
data receiving apparatus. 

Fig. 21 is a flowchart of processing executed by the 
25 data transmission apparatus. 

Fig. 22 is a flowchart of processing executed by the 
data receiving apparatus. 

Fig. 23 is a view of an example system structure of 
a data transmission apparatus and a data receiving 
30 apparatus. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

35 [System outline and data transmission and receiving 
structure example] 

[0075] An outline of a system and an example data 
transmission and receiving structure according to the 

^0 present invention will be described first. An image-infor- 
mation distribution system according to the present in- 
vention executes scalable encoding by using wavelet 
transform. Hierarchical code such as JPEG-2000, 
based on wavelet transform, can have layers where res- 

15 olutions are finely specified, and it is easy to specify lay- 
ers corresponding to any bit rates determined by various 
data receiving terminals having different processing 
performance. Since a JPEG-2000 video stream, which 
is a compression format based on JPEG-2000 for mo- 

50 tion images, is formed of a series of intra-frame data 
without any interframe correlation, even if a packet loss 
occurs on a network, an error is not propagated from the 
lost packet to other packets. Therefore, when the wave- 
let transform is used, since block noise does not occur 

55 visual deterioration of image quality is suppressed. The 
present invention provides structures for scalable-en- 
coding, packetizing. receiving, and decoding used when 
data to which the wavelet transform, which has the 
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above characteristics, is applied is distributed through 
a communication network, such as the Internet. 
[0076] A system according to the present invention, 
to be described below, performs scalable encoding by 
wavelet transform, and transmits and receives scalable- 5 
encoded data. Levels of priority are assigned to data in 
a plurality of layer levels according to an application and 
the network layer, and used. For example, data commu- 
nication is performed with error control or rate control 
being executed according to levels of priority. With this 10 
structure, high-quality image communication is allowed 
in image data transfer on a network where a packet loss 
may occur. 

[0077] Further, when data receiving terminals having 
various, different processing performance refers to lev- is 
els of priority corresponding to layers specified in scal- 
able-encoding processing performed by using wavelet 
transform to select necessary packets only, a data 
source can execute communication suited to the per- 
formance of terminals in heterogeneous environments 20 
to implement a scalable communication system. 
[0078] Fig. 1 shows a data transmission and receiving 
system according to an embodiment of the present in- 
vention. In Fig. 1 , a video camera 11 serves as an input 
apparatus for a data receiving side. The video camera 25 

1 1 is connected to an encoding and communication ap- 
paratus 12 at a data transmission site. In this system, it 
is assumed that networks are connected by the Internet 
Protocol (IP). 

[0079] The encoding and communication apparatus 30 

12 at the data transmission site executes scalable en- 
coding processing based on the waveiet transform, 
packetizes (applies packet generation processing to) 
encoded data in each of a plurality of layers specified 

by the scalable encoding processing, and sends IP 35 
packets generated in this way to a network 13. 
[0080] The network 13 sends the IP packets to a 
transmission destination according to address informa- 
tion specified in the IP packets. Various transmission 
forms are used. For example, packets are sent to a ter- 40 
minal 17 through a service provider network 14 which 
offers dial-up service, packets are sent to a terminal 1 8 
through a service provider network 15 which uses AD- 
SL, or packets are sent to a mobile terminal 1 9 through 
a base station 16 by a radio network. 45 
[0081] Each of the data receiving terminals 17. 18. 
and 19 reports to the encoding and communication ap- 
paratus 12 quality of service (QoS) information, such as 
the information of data rates at which the terminal can 
connect to a network, decoding possibility of encoded 50 
data received in the range of the data rates, and a res- 
olution at which the terminal can display data, and a bit 
rate corresponding to the CPU performance of the ter- 
minal. The encoding and communication apparatus 12 
determines a layer(s) to which information is to be sent, 55 
according to the available bandwidths of the networks 
to be used and the QoS information received from each 
of the data receiving terminals 17, 18, and 1 9 : as termi- 



nal-performance information, and sends necessary 
packets to each terminal. 

[0082] When data is distributed by a unicast method, 
each of the terminals 17, 18, and 19 receives all packets 
sent to the terminal from the encoding and communica- 
tion apparatus 12. When data is distributed by a multi- 
cast method, each terminal receives packets having a 
level of priority equal to or lower than that requested by 
the terminal. The terminals 17. 18, and 19 execute pack- 
et selection processing according to the information of 
the level of priority attached to each packet by the en- 
coding and communication apparatus 12. Details of the 
processing will be described later. 
[0083] The execution of packet selection and receiv- 
ing is effective in the terminals 17. 18, and 1 9 when, for 
example, a band width is provided wider than required 
by terminal performance and topology is used such as 
a LAN in which terminals are connected through a bus. 
Fig. 2 shows example data distribution in a bus-type 
LAN. In Fig. 2, a server 21 performs scalable encoding 
and sends packetized data for each layer through a bus. 
It is assumed that the server 21 outputs packets 22, 23. 
and 24, and they have a first level, a second level, and 
a third level of priority. The levels of priority are assigned 
to layers in a descending order from the most important 
layer. The packet having the first level of priority includes 
encoded data of the most important layer, and the pack- 
et having the third level of priority includes encoded data 
of the least important layer. 

[0084] The encoded data of the most important layer 
is basically required to display images for both of a ter- 
minal having a low-resolution display and a low process- 
ing performance and a terminal having a high-resolution 
display and a high processing performance. The encod- 
ed data of a layer having a low level of priority is effec- 
tive, for example, when a high-quality image data is dis- 
played on a high-resolution display, but is meaningless 
for data display in a low-resolution display. Details will 
be described later. 

[0085] The level of priority of each packet on the bus 
is detected, and only packets required according to the 
performance of each terminal, such as its resolution or 
the processing performance of its CPU, are decoded to 
allow scalable distribution. In Fig. 2, a terminal 25 se- 
lectively receives the packet having the first level of pri- 
ority, a terminal 26 receives the packets having the first 
and second levels of priority, a terminal 27 receives the 
packets having the first, second, and third levels of pri- 
ority, and a terminal 28 receives the packets having the 
first and second levels of priority. In this case, the termi- 
nal 27 has the highest resolution and the highest CPU 
processing performance. Each terminal decodes en- 
coded data included in the packet(s) selectively re- 
ceived to display data. 

[0086] Alternatively, also in a broadcasting system 
which uses a satellite, scalable communication is pos- 
sible when receivers receive only packets suited to their 
performance. Fig. 3 shows an example broadcasting 
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system which uses a satellite. A server 35 executes 
scalable encoding processing based on the wavelet 
transform, packetizes (applies packet generation 
processing to) encoded data in each of a plurality of lay- 
ers specified by the scalable encoding processing, and s 
sends generated packets to a satellite 31 . Each of ter- 
minals 36 to 39 receives packets having the level(s) of 
priority required according to the performance of the ter- 
minal through the satellite 31 . 

[0087] In Fig. 3. the same processing as that shown 10 
in the LAN of Fig. 2 is shown. It is assumed that the 
server 35 outputs packets 32, 33, and 34, and they have 
a first level, a second level, and a third level of priority. 
The packet having the first level of priority includes en- 
coded data of the most important layer, and the packet is 
having the third level of priority includes encoded data 
of the least important layer. 

[0088] The level of priority of each packet is detected, 
and only packets required according to the performance 
of each terminal, such as its resolution or the processing 20 
performance of its CPU : are decoded. In Fig. 3, the ter- 
minal 36 selectively receives the packet having the first 
level of priority, the terminal 37 receives the packets 
having the first and second levels of priority, the terminal 
38 receives the packets having the first, second, and 25 
third levels of priority, and the terminal 39 receives the 
packets having the first and second levels of priority. 
Each terminal decodes encoded data included in each 
packet to display data. 

30 

[Structure and processing of data transmission site] 

[0089] The structure and processing of the data trans- 
mission site in the system according to the present in- 
vention will be described next. The data transmission 35 
site generates encoded data, packetizes the data, and 
sends it, in other words, executes encoding processing ; 
packet generation, and packet transmission processing. 
Fig. 4 is a block diagram of the encoding and commu- 
nication apparatus of the data transmission site in the 40 
system according to the present invention. In Fig. 4, a 
camera 41 is used as an input apparatus of data to be 
encoded. Images and audio signals obtained by the 
camera 41 are input to an encoder 42. Data to be en- 
coded is not limited to data obtained by the camera. It *s 
may be data stored in various storage media, such as 
a DVD, a CD, and a hard disk or data received from other 
servers. 

[0090] The encoder 42 applies the wavelet transform 
as encoding. The encoder 42 executes progressive en- 50 
coding processing in a progressive order specified in ad- 
vance. More specifically, the encoder 42 executes scal- 
able encoding progressively based on spatial resolu- 
tions, signal-to-noise ratios (SNRs), that is, image qual- 
ity, or color components (such as R, G, and B compo- 55 
nents, or Y Cb. and Cr components), and temporarily 
stores encoded data in a buffer 43. 
[0091] Progressive encoding is frequently used for 



image distribution on the Internet, and allows a data re- 
ceiving terminal to display a coarse image first, and then 
gradually finer images. In progressive encoding based 
on spatial resolutions, low-frequency-image data corre- 
sponding to a coarse image is encoded and then high- 
frequency-image data corresponding to finer images is 
encoded. A receiving terminal first decodes the encoded 
low-frequency-image data and displays the coarse, 
rough image on a display within a short period. Then, 
the receiving terminal decodes the encoded high-fre- 
quency-image data and displays finer images gradually. 
In progressive encoding based on SNRs, that is, based 
on image quality, low-SNR (low-quality) data is distin- 
guished from high-SNR (high-quality) data and encod- 
ed. In progressive processing based on color compo- 
nents (R, G : and B components or Y, Cb, and Cr com- 
ponents), each color component is encoded. 
[0092] Fig. 5 shows an example structure of the en- 
coder 42, which executes the wavelet transform. Octave 
division, which is the most general wavelet transform 
among several wavelet transform methods, is applied 
to a plurality of levels. In Fig. 5, the number of levels is 
three (level 1 to level 3), an image signal is divided into 
a low-frequency component and a high-frequency com- 
ponent, and only the low-frequency component is divid- 
ed hierarchically. For convenience, wavelet transform is 
applied to a one-dimensional signal (such as the hori- 
zontal component of an image) in Fig. 5. When the 
wavelet transform is extended two dimensionally, a two- 
dimensional image signal can be handled. 
[0093] The operation of the encoder 42 will be de- 
scribed next. An input image signal 250 to a wavelet 
transform section is divided in band by a low-pass filter 
211 (having a transfer function H0(z)) and a high-pass 
filter 212 (having transfer function H1(z)), and resolu- 
tions in the obtained low-frequency component and 
high-frequency component are halved (level 1 ) by down 
samplers 213 and 214. The outputs thereof are an L 
component 251 and an H component 256, where L in- 
dicates a low frequency, and H indicates a high frequen- 
cy. The low-pass filter 21 1 . the high-pass filter 21 2, and 
the two down samplers 213 and 214 form a level-1 cir- 
cuit section 210 in Fig. 5. 

[0094] Only the low-frequency component signal out- 
put from the down sampler 213. of the signals thinned 
out by the down samplers 213 and 214 is further divided 
in band by a low-pass filter and a high-pass filter of a 
level-2 circuit section 220. The resolutions of the signals 
output from the filters are halved (level 2) by down sam- 
plers. The level-2 circuit section 220, formed of the low- 
pass filter, the high-pass filter, and the down samplers, 
has the same structure as the level-1 circuit section 210. 
formed of the low-pass filter 21 1 , the high-pass filter 21 2, 
and the down samplers 213 and 214. 
[0095] The above processing is performed to a pre- 
determined level to divide in band low-frequency com- 
ponents hierarchically to sequentially generate band 
components. Band components generated in level 2 are 
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an LL component 252 and an LH component 255. Fig. 
5 shows band divisions performed to level 3. The output 
of the down sampler connected to the low-pass filter in 
the level-2 circuit section 220 is sent to a level-3 circuit 
section 230 which has the same structure as the level- 
1 circuit section 210. When band divisions are per- 
formed to level 3 in this way, an LLL component 253, an 
LLH component 254, the LH component 255, and the H 
component 256 are generated. 

[0096] Fig. 6 shows band components obtained when 
a two-dimensional image is divided in band to level 3. 
The notation of "L" and H H" in Fig. 6 differs from that in 
Fig. 5, which handles a one-dimensional signal. In Fig. 
6, a two-dimensional signal is divided in band (horizon- 
tally and vertically) at level 1 into four components LL 
LH, HL, and HH, where LL indicates that horizontal and 
vertical parts are both L. and LH indicates that a hori- 
zontal part is H and a vertical part is L. Then, the LL 
component is further divided in band to generate LLLL. 
LLHL. LLLH. and LLHH components. Further, the LLLL 
component is again divided in band to generate LLLLLL : 
LLLLHL. LLLLLH. and LLLLHH components. 
[0097] The encoder 42 shown in Fig. 4 executes the 
above-described wavelet transform processing. Data 
encoded by the encoder 42 is stored in the buffer 43 at 
each layer level. Layer level divisions based on spatial 
resolutions of data to which the wavelet transform is ap- 
plied will be described by referring to Fig. 7. A structure 
shown in Fig. 7 corresponds to the data structure shown 
in Fig. 6. Fig. 7 shows a case in which a division process 
is performed three times in ihe wavelet transform, as 
described by referring to Fig. 6. 

[0098] The most-important layer level of data is re- 
quired for displaying a coarse and rough image on a dis- 
play : and corresponds to an encoding area which in- 
cludes a low-frequency component (3LL) : that is, data 
areas 701 to 704 having a size of one eighths in total. 
The next-important layer level of data corresponds to 
next-lower-frequency data areas 701 to 707 having a 
size of one fourth in total. The still-next-most-important 
layer level of data corresponds to still-next-lower-fre- 
quency data areas 701 to 71 0 having a half size in total. 
[0099] The data layer-encoded in this way is output to 
and stored in the buffer 43. Fig. 8 shows the structure 
of the data output from the encoder 42. 
[0100] The structure of the data output from the en- 
coder 42, shown in Fig. 8, will be described next. The 
output data starts with a start-of-code-stream (SOC) 
maker indicating the start of code data, a main header 
follows in which an encoding parameter a quantization 
parameter, and a progressive order are described, and 
then encoded data follows. The encoded data has a hi- 
erarchical structure. At the end of the code data, an end- 
of-code-stream (EOC) maker indicating the end of the 
code data is disposed. 

[0101] A packetizer 44 serving as packet generation 
means analyzes the encoded data stored in the buffer 
43, determines partitions according to the contents of 



the data, and packetizes the data. The packetizer 44 us- 
es the main header in the data stored in the buffer 43 to 
obtain the progressive-order information of the encoded 
data, the number of layers, and information related to 

5 color components. The packetizer 44 reads these piec- 
es of field information to analyze the layer structure. Lay- 
er-level structures include a progressive structure 
based on spatial resolutions, a progressive structure 
based on SNRs, that is, image quality, and a progressive 

w structure based on colorcomponents (R, G, and B com- 
ponents or Y, Cb, and Cr components). 
[0102] Packet generation processing and processing 
for attaching levels of priority to packets performed in 
the packetizer 44 will be described next by referring to 

15 Fig. 9. Data is input from the camera 41 to the encoder 
42. The encoder 42 applies scalable encoding to the da- 
ta according to the wavelet hierarchical structure and 
stores encoded data in the buffer 43. The packetizer 44 
reads the main header of the encoded data stored in the 

20 buffer 43 to obtain the progressive order information , the 
number of layers, and information related to color com- 
ponents, partitions the encoded data according to the 
obtained information in each layer, and applies packet 
generation processing to the partitioned data. 

25 [0103] Three example layer structures based on pro- 
gressive orders defined in JPEG-2000 and their corre- 
sponding example packet structures will be described 
next. 

[01 04] Fig. 1 0 is a view showing a case in which data 

30 encoded according to a progressive order of spatial res- 
olutions is associated with packets in each layer. An im- 
age of one frame is formed of packets 801 to 804. When 
encoded data stored in the packet 801 is decoded, an 
image 805 having a spatial resolution one eighth that of 

35 the original is obtained. The packet 801 includes encod- 
ed data having the highest level of priority, and is re- 
quired to display a coarse image first on the display of 
a data receiving terminal which executes decoding. 
When encoded data stored in the packet 801 and the 

40 packet 802 is decoded, an image 806 having a spatial 
resolution one fourth that of the original is obtained. 
When encoded data stored in the packets 801 to 803 is 
decoded, an image 807 having a spatial resolution half 
that of the original is obtained. When encoded data 

45 stored in the packets 801 to 804 is decoded, an image 
808 having the same spatial resolution as the original is 
obtained. 

[0105] Thepriority orderof the fourpackets 801 to804 
shown in Fig. 10 is the packet 801 , the packet 802, the 

50 packet 803, and the packet 804. When scalable-encod- 
ed data is generated by progressive encoding process- 
ing with a progressive order of different spatial resolu- 
tions, as shown in Fig. 10, the packetizer 44 executes 
processing for setting priority-level information corre- 

55 sponding to each spatial-resolution layer of encoded da- 
ta of an image signal, stored in generated packets, as 
additional information of the generated packets. 
[0106] Fig. 11 is a view showing a case in which data 
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encoded according to a progressive order of SNRs (im- 
age quality) is associated with packets in each layer. An 
image of one frame is formed of packets 811 to 814. 
When encoded data stored in the packet 811 is decod- 
ed, an image 815 having the same spatial resolution as 
the original low-quality image is obtained. When encod- 
ed data stored in the packet 811 and the packet 812 is 
decoded, an image 816 having a higher image-quality 
than the decoding result of the encoded data stored in 
the packet 811 is obtained. When encoded data stored 
in the packets 811 to 8 1 3 is decoded . an image 8 1 7 hav- 
ing a further higher image-quality with lower noise is ob- 
tained. When encoded data stored in the packets 811 
to 814 is decoded, an image 81 8 having the highest im- 
age quality is obtained. 

[0107] The priority order of the four packets 811 to 814 
shown in Fig. 11 is the packet 811 . the packet 81 2 : the 
packet 813, and the packet 814. When scalable-encod- 
ed data is generated by progressive encoding process- 
ing with a progressive order of different SNRs, as shown 
in Fig. 11 . the packetizer44 executes processing for set- 
ting priority- level information corresponding to each 
SNR layer of encoded data of an image signal., stored 
in generated packets, as additional information of the 
generated packets. 

[0108] Fig. 1 2 is a view showing a case in which data 
encoded according to a progressive order of color com- 
ponents is associated with packets in each layer. An im- 
age of one frame is formed of packets 821 to 823. When 
encoded data stored in the packet 821 which includes 
Y-component encoded data is decoded, a black-and- 
white image 825 is obtained. When encoded data stored 
in the packet 821 and the packet 822 is decoded, a color 
image 826 which includes a U component is obtained. 
When encoded data stored in the packets 821 to 823 is 
decoded, a color image 827 which includes a V compo- 
nent is obtained. 

[0109] The priority order of the three packets 821 to 
823 shown in Fig. 12 is the packet 821 . the packet 822 : 
and the packet 823. When scalable-encoded data is 
generated by progressive encoding processing with a 
progressive order of different color components, as 
shown in Fig. 12, the packetizer44 executes processing 
for setting priority-level information corresponding to 
each color-component layer of encoded data of an im- 
age signal, stored in generated packets, as additional 
information of the generated packets. 
[0110] The packetizer 44 executes processing for 
generating (IP) packets in which encoded data in each 
layer serves as a payload. A real-time transport protocol 
(RTP) is used as a real-time transmission and receiving 
protocol of image and audio data in an IP network. 
[0111] The packetizer 44 adds an RTP payload head- 
er for a JPEG-2000 video stream to a payload parti- 
tioned in each layer as a part of an RTP header, and 
further adds an RTP fixed header to make a packet. Fig. 
1 3 shows the RTP payload headerfor a JPEG-2000 vid- 
eo stream and the RTP fixed header. A flag indicating a 



level of priority is included in the RTP payload header 
of a packet which accommodates data packet ized in lay- 
er. The level of priority is stored in a [priority] field of the 
RTP payload header. 

5 [0112] When data encoded according to a progres- 
sive order is packetized. if an image of one frame is 
formed of packets 801 to 804 as shown in Fig. 10, the 
highest level of priority is assigned to the packet 801 , 
which generates the image 805 having a spatial resolu- 

10 tion one-eighths that of the original, and levels of priority 
are assigned in a descending order to the packet 802, 
the packet 803, and the packet 804. The level of priority 
is stored in the [priority] field of the RTP payload header 
of each packet. 

'5 [0113] The structure of the RTP payload header 
shown in Fig. 1 3 will be described. A [type] field indicates 
the type of a packet carried by RTP in a JPEG-2000 vid- 
eo stream. To define a packet format other than that 
used in the present case, the value of the field is 

20 changed appropriately. 

[0114] The [priority] field indicates the degree of im- 
portance of the RTP packet as described above, and is 
determined according to an application or service used. 
[01 1 5] An [mhMd] field indicates the identifier of each 

25 JPEG-2000 main header, and is used to prevent the 
packet from being unable to be decoded when a packet 
which includes the main header is lost. A [mhjength] 
field indicates the length of the JPEG-2000 main header, 
and is used as an offset for detecting the loss of the main 

30 header or for detecting packets following the main head- 
er. A [fragment offset] field indicates the offset of the 
packet from the starting byte of the JPEG-2000 data car- 
ried by the RTP packets. 

[01 1 6] The RTP fixed header includes the fields of a 

35 version number (V). padding (P), whether an extension 
header (X) is disposed, the number (CC) of transmission 
sources, marker information (M), a payload type (PT), 
a sequence number, an RTP time stamp, a synchroni- 
zation source (SSRC) identifier, and a contributing 

40 source (CSRC) identifier. The time stamp specified in 
the RTP header is used to control a processing time 
when the RTP packet is developed, allowing the repro- 
duction of a real-time image or sound to be controlled. 
A plurality of scalable-encoded data items, which are 

45 compressed : can be stored in an IP packet. 

[01 17] An IP header is further added to the packet to 
which the RTP header has been added. Fig. 14 shows 
details of the IP header in an IP packet. The IP header 
includes a version, such as IPv4 or IPv6, a header 

50 length, a type-of-service (TOS) field which stores prior- 
ity-level information : a packet length, an identification, 
a flag indicating control information related to data divi- 
sion (fragment) in an IP layer, a fragment offset indicat- 
ing the location of divided (fragmented) data, time to live 

55 (TTL) indicating the information of time until the data is 
discarded, a protocol (IP: 4, TCP: 7, UDP: 17, .) used 
in an upper layer, a header checksum, a source IP ad- 
dress, and a destination IP address. 
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[0118] A flag indicating a level of priority to be proc- 
essed by a receiving side may be added in the IP head- 
er, in addition to the level of priority specified in the RTP 
payload header. For example, in IPv4, a level of priority 
is indicated in the TOS field, and priority control is pos- 
sible for packets having levels of priority in a DiffServ 
network. In IPv6, it is possible to indicate a level of pri- 
ority in a flow label. In this way, when a different protocol 
is used in the network layer, the value indicating a level 
of priority also differs. Therefore., it is preferred that re- 
lationships among encoder layers, levels of priority of 
packets with an application used taken into account, and 
a level of priority in the network layer be specified. A 
control block 45 performs such a control. The packetizer 
44 may execute priority-level setting processing under 
the control of the control block 45 according to a setting 
corresponding to the network protocol. Alternatively, 
levels of priority may be set according to QoS received 
from a receiving terminal or may be dynamically 
changed according to a network condition. 
[0119] IP packets generated by the packetizer 44 are 
output from the communication system through a net- 
work interface 46. DiffServ is service proposed in IETF. 
In DiffServ, a router changes a packet processing meth- 
od according to a level of priority specified in the TOS 
field of an IP header. The objective of this service is to 
increase the quality of the network by placing priority on 
real-time sound and images over usual data in process- 
ing, and discarding a packet having a lower level of pri- 
ority when congestion occurs in a router and packets 
need to be discarded. 

[0120] A method for assigning levels of priority to an 
RTP header which include an RTP payload header and 
to an IP header will be described next. When data en- 
coded according to the progressive order of spatial res- 
olutions shown in Fig. 10 is packetized, a higher level of 
priority is assigned to a packet which includes a lower- 
resolution encoded data, and a lower level of priority is 
assigned to a packet which includes a higher-resolution 
encoded data. When data encoded according to the pro- 
gressive order of SNRs (image quality) shown in Fig. 1 1 
is packetized, a higher level of priority is assigned to a 
packet which includes a lower-SNR (image quality) en- 
coded data, and a lower level of priority is assigned to 
a packet which includes a higher-SNR (image quality) 
encoded data. When data encoded according to the pro- 
gressive order of color components shown in Fig. 12 is 
packetized, a higher level of priority is assigned to a 
packet which includes a Y-component encoded data ; 
and a lower level of priority is assigned to a packet which 
includes a U-component or V-component encoded data. 
[0121] For example, levels of priority are specified by 
the packetizer 44 as shown in Fig. 9. The packetizer 44 
divides data scalable-encoded by the encoder 42 into 
layer-0 to layer-4 encoded data, and stores in five pack- 
ets 65 to 69 as payloads. 

[01 22] Since the layer-0 encoded data is the most im- 
portant, a ievel [0] of priority is assigned to the RTP 



header (RTPH) of the IP packet 65, which has the layer- 
0 encoded data as a payload, and the level [0] of priority 
is assigned to its IP header (IPH). Since the layer- 1 en- 
coded data is the next most important, a level [1] of pri- 

5 ority is assigned to the RTP header (RTPH) of the IP 
packet 66 , which has the layer- 1 encoded data as a pay- 
load, and the ievel [1] of priority is assigned to its IP 
header (IPH). A level [2] of priority is assigned to the 
RTP header (RTPH) of the IP packet 67, which has the 

io layer-2 encoded data as a payload, and the level [1] of 
priority is assigned to its IP header (IPH). A level [3] of 
priority is assigned to the RTP header (RTPH) of the IP 
packet 68, which has the layer-3 encoded data as a pay- 
load, and the level [2] of priority is assigned to its IP 

is header (IPH). A level [4] of priority is assigned to the 
RTP header (RTPH) of the IP packet 69 : which has the 
layer-4 encoded data as a payload, and the level [2] of 
priority is assigned to its IP header (IPH). 
[0123] The priority-level setting processing for IP 

20 headers and RTP headers is, for example, executed ac- 
cording to a priority- level setting map stored in storage 
means of the packetizer 44. Fig. 15 shows an example 
priority- level setting map. In the priority-level setting 
map. levels of priority assigned to an RTP header and 

25 an IP header are associated with each layer of data en- 
coded by the encoder 42. 

[0124] In the example priority-level setting map 
shown in Fig. 15, the levels 0 to 4 of layers set in the 
wavelet transform are used as they are as the levels 0 

30 to 4 of priority assigned to the extension headers (RTP 
payload headers following RTP fixed header) of RTP 
packets. The three levels 0 to 2 of priority are assigned 
to IP headers, sothatthree levels of priority are specified 
when viewed from the IP network. When layer levels are 

35 associated with levels of priority in RTP and the levels 
of priority in RTP are mapped onto those in IP in this 
way, the following control, for example, is made possi- 
ble. 

[0125] Since the sequence numbers of packets are 

40 controlled in RTP : if a packet is lost on the Internet, the 
lost packet can be detected. When the packet loss is 
detected, the receiving side can, for example, report the 
position of the packet loss to a decoder to change an 
error control method. A forward error correction (FEC) 

45 method is, for example, used as an error control method. 
A method for applying FEC to a packet loss in AAL1 of 
ATM or a method similar to a method in which a matrix 
described in the specification of the types 1 and 2 of 
B-ISDN ATM adaptation layer (AAL) in ITU-T Recom- 

50 mendation 1.363.1 is generated to apply Reed-Solomon 
decoding to the lost packet can be used. 
[01 26] When the receiving side detects a packet loss, 
the receiving side can report the lost packet to the data 
transmission side and request the data transmission 

55 side to send again the lost packet. In this case, the re- 
ceiving side can execute recovery processing by obtain- 
ing the lost packet by re-transmission. 
[0127] This type of processing is not necessarily per- 
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formed uniformly for packets of all layer levels. For ex- 
ample, the redundancy of forward error correction may 
be changed according to the bandwidth of a network, or 
the number of times re-transmission is performed may 
be weighted according to a level of priority. 5 
[0128] The packetizer may always use the priority- 
level setting map shown in Fig. 15 in a fixed manner. 
Alternatively, the packetizer may dynamically change 
the priority-level setting map according to a network 
condition. To dynamically change the priority-level set- 
ting map according to a network bandwidth, for exam- 
ple, a method for monitoring a bandwidth at the trans- 
mission side and the receiving side, or a method for 
measuring the loss rate of transmission packets by 
RTCP may be used. Then, mapping can be changed 
with a transmission bandwidth and quality which can be 
assured being taken into account. When a priority-level 
setting map which maps layer levels onto levels of pri- 
ority in RTP packets, or onto levels of priority in IP pack- 
ets is generated, levels of priority can be determined 
with a network loss rate which determines the effect of 
a loss on the original image being taken into account. 
[01 29] As described above, since levels of priority cor- 
responding to the layer levels of encoded data are set 
in the RTP payload headers of RTP headers in the data 
transfer processing according to the present invention, 
a level of priority depending on an upper-layer applica- 
tion can be understood in RTP, and therefore processing 
for a packet loss can be changed. 
[0130] A process for each layer level depends on the 
application used, and a process to be performed accord- 
ing to levels of priority may be determined only in RTP. 
In the same way, the processing method can be 
changed with levels of priority being assigned in IP. In 
this case, since this is a function offered by a network, 
such as DiffServ, in IP, levels of priority in RTP packets 
are mapped onto levels of priority in IP packets to spec- 
ify levels of priority supported by the network, or deter- 
mined by the network. 

[0131] In the priority-level setting map shown in Fig. 
15, five levels of priority are specified in RTP packets, 
and three levels of priority are specified in IP packets. 
The level 0 of priority in RTP packets corresponds to the 
level 0 of priority in IP packets, the levels 1 and 2 of 
priority in RTP packets correspond to the level 1 of pri- 
ority in IP packets, and the levels 3 and 4 of priority in 
RTP packets correspond to the level 2 of priority in IP 
packets. The number of levels of priority which can be 
handled by DiffServ is small in the current IPv4 format, 
but a process corresponding to three levels of priority 
can be performed with the mapping used in the present 
case being applied. 

[0132] When data encoded according to the progres- 
sive order of spatial resolutions shown in Fig. 1 0 is pack- 
etized. for example, a higher level of priority is assigned 
to a packet which includes lower-resolution encoded da- 
ta, and a lower level of priority is assigned to a packet 
which includes higher-resolution encoded data. A re- 



ceiving terminal can use the level of priority assigned to 
the I P header or the RTP header of each packet to proc- 
ess lower-frequency packets earlier. Even when con- 
gestion occurs in a network, the rate of discarding pack- 
ets is reduced and image quality is improved. 
[01 33] Fig. 1 6 shows an example priority-level setting 
map in which levels of priority assigned to the RTP head- 
ers and the IP headers of scalable-encoded data are 
associated with resolutions and SNRs (image quality). 
[0134] In the priority-level setting map shown in Fig. 
1 6, resolutions have three layer levels 0 to 2, and SNRs 
(image quality) have three layer levels 0 to 2. In other 
words, the number of wavelet divisions in JPEG-2000 
is set to two, that is. three levels of resolutions are used, 
and three levels of SNRs (image quality) are used. The 
lower the level number is, the more important the corre- 
sponding data is. 

[01 35] When mapping is performed to the RTP head- 
ers of data encoded according to a progressive order of 
spatial resolutions, for example, since an image having 
a higher spatial resolution is decoded by using an image 
having a lower spatial resolution, it is necessary to as- 
sign a higher level of priority to data having a lower spa- 
tial resolution. In a network where its bandwidth is not 
wide enough to divide image quality into levels, the high- 
est level [0] of priority is assigned to both the RTP head- 
er and the IP header of a packet which includes encoded 
data corresponding to a level [0] of resolution, irrespec- 
tive of image quality. 

[0136] According to this rule, in the present case, 
mapping is performed such that different levels of prior- 
ity are assigned to RTP headers according to two 
groups of SNRs (image quality) for encoded data having 
resolution levels 1 and 2. More specifically, the level [1] 
of priority is assigned to the RTP header of a packet 
which includes encoded data having the resolution level 
1 and the SNR level 0. and the level [2] of priority is as- 
signed to the RTP header of a packet which includes 
encoded data having the resolution level 1 and the SNR 
level 1 or 2. The level [3] of priority is assigned to the 
RTP header of a packet which includes encoded data 
having the resolution level 2 and the SNR level 0 or 1 , 
and the level [4] of priority is assigned to the RTP header 
of a packet which includes encoded data having the res- 
olution level 2 and the SNR level 2. When re-transmis- 
sion is performed with error control, a request for re- 
transmitting a packet having a higher level of priority pre- 
vents congestion of re-transmitted packets. 
[0137] It is preferred that mapping from RTP to IP be 
performed such that levels of priority which can be sup- 
ported by network layer service such as DiffServ are 
specified. In the priority-level setting map shown in Fig. 
16, three levels of priority are assigned to IP headers, 
and the levels of priority for resolutions are mapped as 
they are onto the levels of priority for the IP headers. 
[01 38] Various priority-level setting maps can be used 
in addition to this case. As described above, it is possible 
that levels of priority depending on an application are 
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assigned to RTP payload headers, and levels of priority 
are further assigned to IP headers, according to the de- 
gree of importance of scalable data. With the use of 
these plurality of levels of priority, an error control meth- 
od can be changed for each layer or rate control can be 
applied to each layer. 

[0139] Levels of priority assigned to RTP payload 
headers may be dynamically changed according to an 
application, a user's request, or receiving-terminal infor- 
mation received from a receiving terminal. Levels of pri- 
ority assigned to IP payload headers may be dynami- 
cally changed according to a network condition, such as 
the degree of congestion in a network. The data trans- 
mission apparatus at the data transmission site receives 
receiving-terminal information from a receiving terminal 
has a structure to analyze the information, and also has 
a structure to monitor and analyze a network condition. 
[0140] Since levels of priority can be specified sepa- 
ratelyfor RTP payload headers and IP headers, different 
levels of priority can be used between the network layer 
and a layer dependent on an application, and hence the 
quality of transmission data can be independently con- 
trolled. When a network supports DiffServ, for example, 
if levels of priority are specified in a manner suited to 
DiffServ, the data loss rate of low-frequency parts of an 
image is reduced. 

[0141] The packetizer 44 shown in Fig. 9 packetizes 
data scalable-encoded by the encoder 42 according to 
the above-described processing, for each layer, stores 
in packets, assigns levels of priority corresponding to 
the layers of the scalable-encoded data to the RTP 
headers and to the I P headers to execute I P-packet gen- 
eration processing. The IP packets generated in this 
way are sent to a network through the network interface 
46. 

[Structure and processing of data receiving site] 

[0142] The structure and processing of a data receiv- 
ing site for receiving packets in which encoded data is 
stored as payloads. in the system according to the 
present invention will be described next. The data re- 
ceiving site executes packet receiving, packet process- 
ing, and decoding of the encoded data stored in the 
packets. 

[0143] Fig. 17 is a block diagram of a data receiving 
apparatus (terminal) in the data receiving site in the sys- 
tem according to the present invention. 
[0144] A network interface 51 receives packets, and 
an I P-packet filtering section 52 determines whether the 
packets are to be processed by a receiving side and ap- 
plies filtering to unnecessary packets. For example, fil- 
tering is performed so as to pass packets having levels 
of priority to be processed according to the performance 
of the terminal, to a decoder 57. For example, a terminal 
having a low-resolution display determines packets hav- 
ing higher levels of priority according to the levels of pri- 
ority assigned to the IP headers. Packets having higher 



levels of priority, such as packets having the level 0 or 
1 of priority assigned to the IP headers, are passed to 
the decoder 57, and packets having the levels 2, 3, ... 
of priority are not passed to the decoder 57 and discard- 
5 ed. 

[0145] Then, an RTP-packet-header scanning sec- 
tion 53 analyzes the levels of priority of the filtered pack- 
ets, and determines whether their order is correct and 
whether there is a missing packet, and stores the pack- 
to ets in an RTP buffer 54. The RTP-packet-header scan- 
ning section 53 determines the levels of priority speci- 
fied in the RTP payload headers of the RTP headers, 
selects only packets having the levels of priority to be 
processed by this data receiving apparatus and passes 
is them to the decoder 57, and does not pass packets hav- 
ing equal or lower levels of priority to or than that to be 
processed by this data receiving apparatus but discards 
them. 

[0146] As described above, the IP-packet filtering 

20 section 52 and the RTP-packet-header scanning sec- 
tion 53 function as packet-priority-level-information de- 
termination means for determining whether packets 
have levels of priority equal to or higher than a value 
specified according to the processing ability of the data 

25 receiving apparatus. The decoder 57 decodes only data 
stored in packets having levels of priority equal to or 
higher than a value specified according to the process- 
ing ability of the data receiving apparatus. 
[0147] The RTP-packet-header scanning section 53 

30 further executes packet-loss detection processing ac- 
cording to sequence numbe. 3 in the FTP headers. 
When a packet loss is detected, an error-correction de- 
termination section 54 determines whether a re-trans- 
mission request is issued with the level of priority being 

35 taken into account. When a re-transmission request is 
issued, a report-packet transmission section 55 sends 
a report packet which stores sequence-number informa- 
tion to the data transmission source to report the se- 
quence number of the RTP packet to be re-transmitted. 

40 [0148] When the RTP buffer 54 outputs packet pay- 
loads to the decoder 57. the decoder 57 decodes re- 
ceived data, that is, encoded data stored as the pay- 
loads of packets, and outputs an image or sound to an 
output apparatus such as a display. 

45 [0149] Fig. 1 8 shows an example detailed structure of 
the decoder 57. The decoder 57 has an inverse wavelet 
transform processing structure for performing the in- 
verse operation of the wavelet transform processing de- 
scribed before by referring to Fig. 5. More specifically, 

so when the bandwidth components 253, 254, 255, and 
256 outputfrom the wavelet transform section described 
by referring to Fig. 5 are input to an inverse wavelet 
transform section shown in Fig. 18, the resolutions of 
the LLL component 253 and the LLH component 254 

55 are doubled by up-samplers 272 and 273. Then, the low- 
frequency component is filtered by a low-pass filter 274, 
the high-frequency component is filtered by a high-pass 
fitter 275, and they are synthesized by an adder. A circuit 
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section 270 having the devices described so far com- 
pletes the inverse transform processing of the transform 
processing performed by the level-3 circuit section 230 
shown in Fig. 5, and an LL component 257 which is a 
level-2 low-frequency component is obtained. This 5 
processing is repeated to the level 1 to output a decoded 
image 259 obtained after final inverse transform. More 
specifically, a level-2 circuit section 280 and a level-1 
circuit section 290 have the same structure as the level- 
3 circuit section 270, the output of the level-3 circuit sec- 10 
tion 270 is input to a low-frequency side of the level-2 
circuit section 280, and the output of the level-2 circuit 
section 280 is input to a low-frequency side of the level- 
1 circuit section 290. The basic structure of a general 
inverse wavelet transform section has been described, '5 
[0150] A re-transmission-request-processing se- 
quence executed when the data receiving side detects 
a packet loss will be described next by referring to Fig. 
19. Since RTP headers have sequence numbers : the 
data receiving site detects a packet loss according to 20 
the RTP headers. The effect of a packet loss on image 
quality depends on the degree of importance of the lost 
packet. For example, when high-frequency encoded da- 
ta is lost, image quality does not largely reduced even 
if the data is not used. When low-frequency data is lost 25 
a large effect is given. 

[0151] The data receiving site performs error control 
according to the degree of importance of a lost packet. 
A forward errorcorrection (FEC) method is. for example, 
used as an error control method. A method for applying 30 
FEC to a packet loss in AAL1 of ATM or a method similar 
to a method in which a matrix described in the specifi- 
cation of the types 1 and 2 of B-ISDN ATM adaptation 
layer (AAL) in ITU-T Recommendation 1.363.1 is gener- 
ated to apply Reed-Solomon decoding to the lost packet 35 
can be used. 

[01 52] The data receiving site changes a re-transmis- 
sion control method according to the degree of impor- 
tance of a lost packet. As shown at an upper part of Fig. 
19, the data transmission site distributes images and 40 
sound by RTP, and the data receiving site detects a lost 
packet according to sequence numbers in RTP packet 
headers, and reports to the data transmission site re- 
transmission request information of the lost packet by a 
packet used in RTCP. The data transmission site re- 45 
transmits the lost packet reported by the data receiving 
site. Before executing re-transmission processing, the 
data transmission site performs control such that the 
level of priority of the packet for which a re-transmission 
request has been issued is determined by using the IP so 
header or the RTP payload header, a packet having a 
higher level of priority is re-transmitted by priority, and 
a packet having a lower level of priority may be not re- 
transmitted in some cases. A re-transmission packet is 
sent together with usual requested data. Therefore : 55 
when a transmission band is limited . control may be per- 
formed such that packet data having a lower level of pri- 
ority is not sent. 



[0153] The data receiving site can receive and proc- 
ess the re-transmitted packet to recover the lost packet 
to increase image quality. The relationships between the 
re-transmission control and levels of priority are shown 
at a lower part of Fig. 19. Error control is enhanced and 
a re-transmission request is issued for data having a 
higher-level (priority level 0) of priority, such as layer- 
level-0 data, and error control is weakened and a re- 
transmission request is not issued for data having a low- 
er level (priority level 2) of priority. Such control can be 
changed according to a network condition and terminal 
processing. 

[0154] When many packet losses occur, a packet re- 
questing re-transmission and a re-transmitted packet 
are also lost in many cases. If re-transmission process- 
ing is performed for all lost packets, packet congestion 
increases, even the recovery of important low-frequen- 
cy data becomes difficult, and image quality deteriora- 
tion cannot be prevented. In contrast, when re-transmis- 
sion control is executed with the quality to be assured 
being determined according to the levels of priority 
stored in RTP payload headers, that is the degrees of 
importance of an image., in other words, when a re-trans- 
mission request is issued only for packets having higher 
levels of priority, it is possible that an increase of packet 
congestion is suppressed, and a higher probability of re- 
ceiving packets having higher degrees of importance, 
for which a re-transmission request has been issued is 
obtained. This method allows important data to be pos- 
itively re-transmitted and image quality to be improved 
even when the same band is used. 

[Processes at data transmission and receiving sites] 

[01 55] Processes performed at the data transmission 
site and the data receiving site will be described below 
together. An example protocol sequence between the 
data transmission side and the data receiving side will 
be described first by referring to Fig. 20. The data re- 
ceiving side issues a setup request to the data transmis- 
sion side by RTSP. The setup request includes the dis- 
play resolution, the CPU processing performance, the 
service quality request, and the usable bandwidth of the 
data receiving side. 

[01 56] When the transmission side can respond to the 
setup request, it responds to the request. Then, the re- 
ceiving side issues a play request by RTSP. When the 
transmission side can respond to the play request, it re- 
sponds to the request, stores scalable-encoded data as 
payloads. generates RTP packets in which levels of pri- 
ority corresponding to layers have been specified in 
RTP payload headers and IP headers, and sends the 
packets. 

[0157] The transmission side sends to the receiving 
side at a constant interval a sender report which in- 
cludes a time stamp and the number of transmitted 
packets according to RTCP. The receiving side receives 
the sender report and sends a receiver report which in- 
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eludes information of the number of lost packets and the 
sequence number of the lost packets, used for re-trans- 
mission processing for the lost packets. The transmis- 
sion side receives the receiver report, performs re-trans- 
mission control, and sends the lost packets. 
[0158] The data transmission side performs control 
such that packets having higher levels of priority are re- 
transmitted earlier with the priority levels of the packets 
being taken into account, as described above. Re-trans- 
mitted packets are sent together with usual requested 
data. Therefore, when a transmission bandwidth is lim- 
ited, control is performed such that packets having lower 
levels of priority in the image-quality progressive order 
are not transmitted. 

[01 59] Processing in the data transmission site will be 
described next by referring to a flowchart shown in Fig. 
21 . A case will be taken as an example, in which scala- 
ble-encoded data is stored in a recording medium in ad- 
vance, and the data is read, packetized, and transmit- 
ted. 

[0160] The transmission apparatus is activated, and 
the preparation of transmission is started. Then, in step 
S101, the transmission side receives a setup request 
from the receiving side by RTSP. When the preparation 
of transmission has been finished, the transmission side 
sends a setup response to the receiving side in step 
S102. 

[01 61] In step S1 03, the transmission side determines 
a priority level and a mapping method for RTP/IP head- 
ers with the display resolution, the CPU processing per- 
formance, the requested service quality, and the usable 
bandwidth of the receiving side being used as input pa- 
rameters. The value is used as an initial value in an RTP- 
priority-level mapping information generation process in 
step S110. 

[0162] In step S104, the transmission side detects a 
packet partition of stored data corresponding to the 
quality requested by the receiving side. The word "pack- 
et" used here refers to a cluster unit in each layer level, 
for example, the minimum code string used in JPEG- 
2000 employing the wavelet transform. In step S105 : 
whether all data has been processed is determined. 
When it has been processed; the processing is termi- 
nated in step S1 13. When it has not yet been processed, 
a priority level corresponding to the layer level of scal- 
able-encoded data stored in the packet is assigned to 
the RTP extension header that is. the RTP payload 
header, in step S106. 

[01 63] The priority level assigned to the RTP payload 
header in step S106 may be that specified in a priority- 
level setting map (such as the map shown in Fig. 15). 
which serves as mapping information specified in ad- 
vance. Alternatively, the priority level may be specified 
according to a priority-level setting map generated in 
S 1 1 0 when a dynamic generation process thereof is ex- 
ecuted according to a network congestion state ob- 
tained in step S109 according to RTCP feedback infor- 
mation. 



[01 64] The data in which the level of priority has been 
assigned to the RTP payload header in step S106 is 
stored in step S1 14 for an RTP re-transmission process. 
To respond to a re-transmission request for a packet 

5 loss, sent from the receiving side, transmission packets 
are stored in step S114 for a predetermined period. A 
packet to be re-transmitted is determined in step S114 
when the number of a lost packet is reported according 
to the RTCP feedback in step S1 09, and a memory area 

to for packets which have been received can be released. 
The re-transmission packet is output in step S124, and 
the processing returns to step S104. 
[0165] In the packet re-transmission process, control 
is performed as described above such that packets hav- 

15 ing higher levels of priority are re-transmitted earlier with 
the levels of priority of packets being taken into account. 
When a transmission bandwidth is limited, for example, 
control is performed such that packets having lower lev- 
els of priority in the image-quality progressive order are 

20 not transmitted. 

[01 66] In step S1 07, an IP-packet generation process 
is executed. A level of priority corresponding to the layer 
of scalable-encoded data stored in an IP packet is spec- 
ified in the IP header. The priority level assigned to the 

25 |p header in step S 1 07 may be that specified in a prior- 
ity-level setting map (such as the map shown in Fig. 15), 
which serves as mapping information specified in ad- 
vance. Alternatively, the priority level may be specified 
according to a priority-level setting map generated in 

30 S112 when a dynamic generation process thereof is ex- 
ecuted according to a network congestion state ob- 
tained in step S111 . The level of priority is specified, for 
example, in the TOS field for IPv4 and in the fiow label 
in IPv6 in the IP header. 

35 [0167] Then, the destination of the IP packet is deter- 
mined in steps S115, S116, and S122. It is determined 
in step S1 1 5 whether unicast is to be performed. When 
it is determined that unicast is to be performed, destina- 
tion information is added in step S11 7. When it is deter- 

40 mined in step S116 that multicast is to be performed, it 
is determined in step S118 that the packet belongs to 
which multicast group according to the level of priority 
assigned to the IP header, a copy of a packet section 
which satisfies the request of the corresponding multi- 

45 cast group is made in step S119, and a multicast ad- 
dress is assigned in step S1 20. The steps from S1 1 8 to 
S120 are repeated the same number of times as the 
number of destinations included in the multicast group, 
and the packet is transmitted in step S121 . 

50 [0168] When it is determined in step S 122 that broad- 
cast is to be performed, a broadcast address is assigned 
in step S123. Addresses other than those described 
above usually are not generated. When the result of the 
determination of step S122 is negative, an error indica- 

55 tion is output in step S125. When the packet is output 
in step S124 : the processing returns to step S104 for 
detecting a packet partition. 

[01 69] As described above, the data transmission site 
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stores scalable-encoded data in a packet to be trans- 
mitted by unlcast. multicast, or broadcast, as a payload, 
and assigns levels of priority corresponding to the layer 
level of the encoded data stored in the packet to the RTP 
payload header and to the IP header. The data trans- 5 
mission site also executes the re-transmission process- 
ing in response to a re-transmission request sent from 
the data receiving site. 

[01 70] Processing in the data receiving site will be de- 
scribed next by referring to a flowchart shown in Fig. 22. 10 
After the preparation of receiving is started, the receiv- 
ing side sends a setup request by RTSP to the trans- 
mission side in step S201. The request includes infor- 
mation such as the performance of a terminal and re- 
quested service quality. The receiving side receives a 15 
setup response from the data transmission side in step 
S202 5 and opens a port for receiving. 
[0171] It is determined in step S203 whether commu- 
nication has been finished. When it is determined that 
the communication has been finished, the processing is 20 
terminated. In step S204, an IP filtering process is exe- 
cuted. It is determined according to the IP header of a 
packet whether the packet is to be processed by the re- 
ceiving side, and an unnecessary packet is filtered. For 
example, this is a filtering process for passing only pack- 25 
ets having levels of priority to be processed according 
to the performance of a terminal to the decoder. When 
the terminal has a low-resolution display, for example : 
packets having higher levels of priority, such as the lev- 
els 0 and 1 of priority, are determined according to the 30 
levels of priority specified in the IP headers and passed 
to the decoders, and packets having the level 2 or larger 
of priority are not passed to the. decoder but discarded. 
[0172] An RTP-packet-header scanning process is 
applied in step S205 to the packets to which filtering has 35 
been executed in step S204 according to the level of 
priority assigned to the IP header. In the RTP-packet- 
header scanning process, the level of priority specified 
in the RTP payload header of the RTP header is deter- 
mined, and only a packet having a level of priority to be 40 
processed by the receiving site itself is passed to the 
decoder. To this end, the packet is stored in a packet 
receiving buffer in step S206. A packet having a low lev- 
el to priority lower than a level of priority to be processed 
by the receiving site itself is not passed to the decoder 45 
but discarded. 

[0173] Then, in step S207 : the RTP header of the re- 
ceived packet is read and it is determined whether its 
sequence number is correct, that is : whether the order 
of the packet is correct and whether a packet has been 50 
lost. When its sequence number is wrong, a packet loss 
is detected in step S208. When a packet loss is found, 
re-transmission control is performed in step S210, and 
the sequence number of the lost packet is reported to 
the transmission side. 55 
[0174] When a packet loss is not found, it is deter- 
mined in step S209 according to the sequence number 
of the RTP header that a sequence error has occurrea. 



When a sequence error has occurred, the order is ar- 
ranged in step S211 . In other cases, an error indication 
is output in step S212. When it is determined in step 
S207 that the sequence number is correct, the packet 
is sent to the decoder in step S213.. and the memory 
area of the sent packet in the packet receiving buffer is 
released. 

[01 75] In step S21 4, a packet statistical process is ex- 
ecuted according to the packet output from the buffer to 
the decoder and the result of the re-transmission proc- 
ess. An RTCP receiver report is sent to the transmission 
side in step S215. 

[01 76] As described above, the data receiving site de- 
termines the level of priority corresponding to the layer 
level of encoded data stored in a received packet ac- 
cording to the RTP payload header and the IP header 
of the received packet, determines whether the decod- 
ing process is applied, and if necessary, performs de- 
coding. In addition, the data receiving site detects a 
packet loss according to the sequence number of the 
RTP packet, and executes error control and re-trans- 
mission control based on the level of priority determined 
according to the RTP payload header and the IP header. 

[Example structure of a data transmission and receiving 
apparatus] 

[0177] The series of processing described in the 
above embodiment can be executed by hardware, by 
software or by a combination of both. When the series 
of processing is executed by software, a program having 
a processing sequence is installed into a memory of a 
data processing apparatus which is built in special hard- 
ware, or into a general-purpose computer, which can ex- 
ecute various functions, and executed. When the series 
of processing is executed by software, a program con- 
stituting the program is installed, for example, into a gen- 
eral-purpose computer or a microcomputer. 
[0178] Fig. 23 shows an example system structure of 
a data transmission apparatus and a data receiving ap- 
paratus which execute the series of processing de- 
scribed in the above embodiment. Data transmitted and 
received in the system of the present invention is scal- 
able-encoded data. The data transmission apparatus 
executes encoding processing, and the data receiving 
apparatus performs decoding processing. Encoded da- 
ta is transmitted and received as packets through a net- 
work. The data transmission side generates packets 
(performs packetizing) and the data receiving side de- 
velops the packets (performs depacketizing). 
[0179] A data transmission and receiving apparatus 
(such as a PC) 850 shown in Fig. 23 includes a codec 
851 for executing encoding, decoding, packet genera- 
tion, and packet development, a network interface 852 
serving as an interface with a communication network, 
an input and output interface 853 serving as an interface 
with input units such as a mouse 837 and a keyboard 
836. an audio and visual interface 854 for inputting and 
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outputting data to and from audio-and-visual-data input 
and output units, such as a video camera 833, a micro- 
phone 834 : and a speaker 835. a display interface 855 
serving as a data output interface with a display 832. a 
CPU 856 for controlling data transfer among the data 
input and output interfaces, the codec 851 , and the net- 
work interface 852, and for controlling various pro- 
grams, a memory 857 formed of a RAM and a ROM, for 
storing various programs controlled by the CPU 856, for 
storing data, and for serving as a working area of the 
CPU 856, and an HDD 858 serving as a medium for stor- 
ing data and programs. All circuit components are con- 
nected through a PCI bus 859, and can allow data to be 
transmitted and received to and from each other. 
[0180] The codec 851 receives, for example, image 
data from the video camera 833 and audio data from the 
microphone 834, and executes scalable encoding and 
packet generation (packetizing) to finally generate IP 
packets having scalable-encoded data as payloads, as 
shown in Fig. 23. The generated IP packets are output 
to the PC I bus 859, output to a network through the net- 
work interface 852, and distributed, for example, to a 
destination address specified in the headers of the IP 
packets. 

[0181] The data transmission and receiving appara- 
tus may be configured such that video data input from 
the video camera 833 and audio data input from the mi- 
crophone 834 are scalable-encoded and output to a net- 
work through the network interface 852 under the con- 
trol of the CPU 856 according to a software encoding 
program stored in the HDD 858 or the memory 857. 
[0182] Data in IP packets input through a network is 
sent to the PCI bus 859 through the network interface 
852 and input to the codec 851 . The codec 851 applies 
packet development processing (depacketizes) the in- 
put data to extract scalable data stored in the packets 
as payloads, executes decoding, and reproduces the 
data to output to the display 832 and the speaker 835. 
[0183] Data to be processed in the above embodi- 
ment, such as images, can be input from an input unit 
such as a camera, from a data input apparatus such as 
a scanner, or from a removable recording medium such 
as a floppy disk, a compact disc read only memory 
(CD-ROM), a digital versatile disk (DVD), a magnetic 
disk, or a semiconductor memory. 
[0184] The CPU 856 can load a program stored in the 
hard disk or a program transmitted from a satellite or a 
network, received, and installed, into a memory such as 
a RAM and execute it. in addition to a program stored 
in the ROM. 

[0185] In the present specification, a program may be 
processed by one computer or may be distributed-proc- 
essed by a plurality of computers. In addition, a program 
may be transmitted from a remote computer and exe- 
cuted. 

[0186] The present invention has been described in 
detail by referring to the specific embodiment. It is clear- 
ly understood that persons skilled in the art can modify 



or change the embodiment within the scope of the 
present invention. In other words, the present invention 
has been disclosed by the use of an example, and 
should not be interpreted in a limited manner. To under- 

5 stand the gist of the present invention . Claims described 
below should be carefully examined. 
[0187] As described above, according to a structure 
of the present invention, since a data transmission site 
stores scalable-encoded data as payloads in packets to 

w be sent, and assigns priority-level information corre- 
sponding to the layer level of the encoded data stored 
in the packets to the packets, a data receiving site exe- 
cutes processing with the priority-level information be- 
ing taken into account, and the most appropriate packet 

is processing corresponding to the performance of a ter- 
minal is allowed. 

[0188] In addition, according to a structure of the 
present invention, since re-transmission control is per- 
formed according to priority- level information corre- 

20 sponding to the layer level of encoded data, encoded 
data having a higher level of priority can be re-transmit- 
ted earlier to suppress the degree of congestion on a 
network. In this case, the arrival rate of re-transmitted 
packets is increased and the quality of data displayed 

25 on a receiving terminal is improved. 

[01 89] Further, according to a structure of the present 
invention, since a level of priority depending on an ap- 
plication is specified in the RTP payload header of a 
packet and a level of priority is also specified in the IP 

30 header according to the degree of importance of scala- 
ble data stored in the packet, an error control method 
can be changed in each layer by using a plurality of piec- 
es of priority-level information, and a rate control can be 
executed. 

35 [0190] Still further according to a structure of the 
present invention, a level of priority specified in an RTP 
payload header is dynamically changed according to an 
application or a user's request. A level of priority speci- 
fied in an IP header is dynamically changed according 

40 to the degree of congestion on a network. Therefore, 
different levels of priority can be used between the net- 
work layer and a layer dependent on an application. The 
quality of transmission data can be independently con- 
trolled. When a network supports DiffServ, for example, 

45 if a level of priority is specified in an IP header in a man- 
ner suitable for DiffServ, a data-loss rate in a low-fre- 
quency part of an image is reduced. 



1. A data communication system comprising a data 
transmission apparatus and a data receiving appa- 
ratus, 

the data transmission apparatus comprising: 

encoding means for applying scalable-encod- 
ing processing by using wavelet transform to an 
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image signal; and 

packet generation means for generating a 
packet which stores scalable-encoded data 
generated by the encoding means and for 
specifying priority-level information corre- 
sponding to a layerof the scalable-encoded da- 
ta of the image signal, stored in the generated 
packet, as additional information of the gener- 
ated packet, and 

the data receiving apparatus comprising de- 
coding means for applying decoding processing to 
data stored in a packet to which the information of 
a level of priority equal to or larger than a value 
specified according to the processing performance 
of the data receiving apparatus is assigned. 

2. A data communication system according to Claim 1 , 

wherein the data receiving apparatus further 
comprises packet-priority-level-infoimation deter- 
mination means for determining whether a packet 
has the information of a level of priority equal to or 
larger than the value specified according to the 
processing performance of the data receiving ap- 
paratus, and 

the decoding means applies the decoding 
processing to data stored in a packet selected by 
the packet-priority-level-information determination 
means. 

3. A data communication system according to Claim 
1 , wherein the packet generation means specifies 
the priority-level information corresponding to the 
layer of the scalable-encoded data of the image sig- 
nal, in an RTP header. 

4. A data communication system according to Claim 
1. wherein the packet generation means specifies 
the priority-level information corresponding to the 
layer of the scalable-encoded data of the image sig- 
nal, in an IP header. 

5. A data communication system according to Claim 
1 , wherein the packet generation means specifies 
the priority- level information corresponding to the 
layer of the scalable-encoded data of the image sig- 
nal, in an RTP header and an IP header. 

6. A data communication system according to Claim 
1. wherein the packet generation means specifies 
the priority-level information corresponding to the 
layer of the scalable-encoded data of the image sig- 
nal, stored in the generated packet, as additional 
information of the generated packet, according to a 
priority-level setting map in which the layer of the 
scalable-encoded data is associated with a level of 
priority. 



7. A data communication system according to Claim 1 , 

wherein the data transmission apparatus fur- 
ther comprises a structure for receiving receiving- 
terminal information from the data receiving appa- 
5 ratus, and 

the packet generation means dynamically 
changes the priority-level information correspond- 
ing to the layer of the scalable-encoded data of the 
image signal, stored in the packet, according to the 
10 receiving-terminal information. 

8. A data communication system according to Claim 
1 , wherein the data transmission apparatus dynam- 
ically changes the priority-level information corre- 

15 sponding to the layer of the scalable-encoded data 
of the image signal . stored in the packet, according 
to a network condition. 

9. A data communication system according to Claim 1 , 
20 wherein the encoding means of the data 

transmission apparatus applies the scalable-en- 
coding processing by using the wavelet transform 
to the image signal, and generates encoded data 
having different spatial resolutions generated by the 

25 wavelet transform, and 

the packet generation means specifies prior- 
ity-level information corresponding to layers of the 
spatial resolutions of the encoded data having the 
different spatial resolutions, as additional informa- 

30 tion of the generated packet. 

10. A data communication system according to Claim 1 , 

wherein the encoding means of the data 
transmission apparatus generates scalable-encod- 

35 ed data by progressive encoding processing, and 

the packet generation means specifies prior- 
ity-level information corresponding to a layer spec- 
ified for a progressive order of the scalable-encod- 
ed data generated by applying the progressive en- 

40 coding processing to the image signal and stored in 

the generated packet, as additional information of 
the generated packet. 

11 . A data communication system according to Claim 1 , 
45 wherein the encoding means of the data 

transmission apparatus generates scalable-encod- 
ed data by progressive encoding processing in 
which different spatial resolutions are specified in a 
progressive order and 

so the packet generation means specifies prior- 

ity-level information corresponding to a layer spec- 
ified for each spatial resolution of the scalable-en- 
coded data of the image signal, stored in the gen- 
erated packet, as additional information of the gen- 

55 erated packet. 

12. A data communication system according to Claim 1 , 

wherein the encoding means of the data 
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transmission apparatus generates scalable-encod- 
ed data by progressive encoding processing in 
which different SNRs are specified in a progressive 
order, and 

the packet generation means specifies prior- 5 
ity-level information corresponding to a layer spec- 
ified for each SNR of the scalable-encoded data of 
the image signal, stored in the generated packet, 
as additional information of the generated packet. 

10 

1 3. A data communication system according to Claim 1 . 

wherein the encoding means of the data 
transmission apparatus generates scalable-encod- 
ed data by progressive encoding processing in 
which different color components are specified in a is 
progressive order, and 

the packet generation means specifies prior- 
ity-level information corresponding to a layer spec- 
ified for each color component of the scalable-en- 
coded data of the image signal, stored in the gen- 20 
erated packet, as additional information of the gen- 
erated packet. 

14. A data communication system according to Claim 

1 , wherein the data transmission apparatus further 25 
comprises a re-transmission control structure for, in 
response to a packet re-transmission request sent 
from the data receiving apparatus, determining the 
levels of priority assigned to packets for which the 
re-transmission request has been sent; for deter- 30 
mining whether re-transmission is to be performed 
according to the determined levels of priority, and 
for re-transmitting packets having higher levels of 
priority earlier. 

35 

15. A data transmission apparatus for storing image da- 
ta in a packet and for transmitting the packet, com- 
prising: 



15, wherein the packet generation means specifies 
the priority-level information corresponding to the 
layer of the scalable-encoded data of the image sig- 
nal, in an IP header. 

18. A data transmission apparatus according to Claim 
15, wherein the packet generation means specifies 
the priority-level information corresponding to the 
layer of the scalable-encoded data of the image sig- 
nal, in an RTP header and an IP header. 

19. A data transmission apparatus according to Claim 
15, wherein the packet generation means specifies 
the priority-level information corresponding to the 
layer of the scalable-encoded data of the image sig- 
nal, stored in the generated packet, as additional 
information of the generated packet, according to a 
priority- 1 eve I setting map in which the layer of the 
scalable-encoded data is associated with a level of 
priority. 

20. A data transmission apparatus according to Claim 
1 5 5 wherein the data transmission apparatus further 
comprises a structure for receiving receiving-termi- 
nal information from the data receiving apparatus, 
and 

the packet generation means dynamically 
changes the priority-level information correspond- 
ing to the layer of the scalable-encoded data of the 
image signal, stored in the packet, according to the 
receiving-terminal information. 

21. A data transmission apparatus according to Claim 
15, wherein the data transmission apparatus dy- 
namically changes the priority-level information cor- 
responding to the layer of the scalable-encoded da- 
ta of the image signal, stored in the packet, accord- 
ing to a network condition. 



encoding means for applying scalable-encod- 40 
ing processing by using wavelet transform to an 
image signal; and 

packet generation means for generating a 
packet which stores scalable-encoded data 
generated by the encoding means and for 45 
specifying priority-level information corre- 
sponding to a layer of the scalable-encoded da- 
ta of the image signal, stored in the generated 
packet : as additional information of the gener- 
ated packet. . 50 

16. A data transmission apparatus according to Claim 
15. wherein the packet generation means specifies 
the priority-level information corresponding to the 
layer of the scalable-encoded data of the image sig- 55 
nal, in an RTP header. 

17. A data transmission apparatus according to Claim 



22. A data transmission apparatus according to Claim 
15, 

wherein the encoding means applies the scal- 
able-encoding processing by using the wavelet 
transform to the image signal, and generates en- 
coded data having different spatial resolutions gen- 
erated by the wavelet transform, and 

the packet generation means specifies prior- 
ity-level information corresponding to layers of the 
spatial resolutions of the encoded data having the 
different spatial resolutions, as additional informa- 
tion of the generated packet. 

23. A data transmission apparatus according to Claim 
15, 

wherein the encoding means generates scal- 
able-encoded data by progressive encoding 
processing, and 

the packet generation means specifies prior- 
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ity-level information corresponding to a layer spec- 
ified for a progressive order of the scalable-encod- 
ed data generated by applying the progressive en- 
coding processing to the image signal and stored in 
the generated packet, as additional information of 
the generated packet. 

24. A data transmission apparatus according to Claim 
15. 

wherein the encoding means generates scal- 
able-encoded data by progressive encoding 
processing in which different spatial resolutions are 
specified in a progressive order, and 

the packet generation means specifies prior- 
ity-level information corresponding to a layer spec- 
ified for each spatial resolution of the scalable-en- 
coded data of the image signal stored in the gen- 
erated packet, as additional information of the gen- 
erated packet. 

25. A data transmission apparatus according to Claim 
15, 

wherein the encoding means generates scal- 
able-encoded data by progressive encoding 
processing in which different SNRs are specified in 
a progressive order, and 

the packet generation means specifies prior- 
ity-level information corresponding to a layer spec- 
ified for each SNR of the scalable-encoded data of 
the image signal, stored in the generated packet, 
as additional information of the generated packet. 

26. A data transmission apparatus according to Claim 
15. 

wherein the encoding means generates scal- 
able-encoded data by progressive encoding 
processing in which different color components are 
specified in a progressive order, and 

the packet generation means specifies prior- 
ity-level information corresponding to a layer spec- 
ified for each color component of the scalable-en- 
coded data of the image signal, stored in the gen- 
erated packet, as additional information of the gen- 
erated packet. 

27. A data transmission apparatus according to Claim 
15. wherein the data transmission apparatus further 
comprises a re-transmission control structure for, in 
response to a packet re-transmission request sent 
from the data receiving apparatus, determining the 
levels of priority assigned to packets for which the 
re-transmission request has been sent; for deter- 
mining whether re-transmission is to be performed 
according to the determined levels of priority, and 
for re-transmitting packets having higher levels of 
priority earlier. 

28. A data receiving apparatus for receiving packets 



which store encoded data, comprising: 

receiving means for receiving packets which 
store data scalable-encoded by wavelet trans- 

5 form: 

packet-priority-level-information determination 
means for determining whether the packets 
have the information of a level of priority equal 
to or larger than a value specified according to 

10 the processing performance of the data receiv- 

ing apparatus; and 

decoding means for decoding data stored in 
packets selected by the packet -priority-level- 
information determination means. 

15 

29. A data receiving apparatus according to Claim 28. 

wherein the packet-priority-level-information 
determination means comprises: 

20 IP-packet filtering means for determining a lev- 

el of priority assigned to the IP header of a re- 
ceived packet; and 

RTP-packet-header scanning means for deter- 
mining a level of priority assigned to the RTP 
25 header of the received packet. 

30. A data communication method for a data communi- 
cation system comprising a data transmission ap- 
paratus and a data receiving apparatus, comprising 

30 the steps of: 

an encoding step of applying scalable-encod- 
ing processing by using wavelet transform to an 
image signal; 

35 a packet generation step of generating a packet 

which stores scalable-encoded data generated 
in the encoding step and of specifying priority- 
level information corresponding to a layer of the 
scalable-encoded data of the image signal, 
*o stored in the generated packet, as additional in- 

formation of the generated packet, both steps 
being executed in the data transmission appa- 
ratus, and 

a decoding step of decoding data stored in a 
45 packet to which the information of a level of pri- 

ority equal to or larger than a value specified 
according to the processing performance of the 
data receiving apparatus is assigned, the de- 
coding step being executed in the data receiv- 
50 ing apparatus. 

31. A data communication method according to Claim 
30, 

further comprising a packet-priority-level-in- 
55 formation determination step of determining wheth- 

er a packet has the information of a level of priority 
equal to or larger than the value specified according 
to the processing performance of the data receiving 



20 



25 



40 



45 



50 



22 



43 



EP 1 311 125 A2 



44 



apparatus, the packet-priority-level-information de- 
termination step being executed by the data receiv- 
ing apparatus, and 

the decoding step decodes data stored in a 
packet selected by the packet-priority-level-infor- 
mation determination step. 

32. A data communication method according to Claim 
30. wherein the packet generation step comprises 
a step of specifying the priority-level information 
corresponding to the layer of the scalable-encoded 
data of the image signal, in an RTP header. 

33. A data communication method according to Claim 
30. wherein the packet generation step comprises 
a step of specifying the priority-level information 
corresponding to the layer of the scalable-encoded 
data of the image signal, in an IP header. 

34. A data communication method according to Claim 
30. wherein the packet generation step comprises 
a step of specifying the priority-level information 
corresponding to the layer of the scalable-encoded 
data of the image signal, in an RTP header and an 
IP header. 

35. A data communication method according to Claim 
30. wherein the packet generation step comprises 
a step of specifying the priority-level information 
corresponding to the layer of the scalable-encoded 
data of the image signal, stored in the generated 
packet, as additional information of the generated 
packet, according to a priority-level setting map in 
which the layer of the scalable-encoded data is as- 
sociated with a level of priority. 

36. A data communication method according to Claim 
30. 

further comprising a step of receiving receiv- 
ing-terminal information from the data receiving ap- 
paratus, the step being executed by the data trans- 
mission apparatus, 

wherein the packet generation step compris- 
es a step of dynamically changing the priority-level 
information corresponding to the layer of the scala- 
ble-encoded data of the image signal, stored in the 
packet, according to the receiving-terminal informa- 
tion. 

37. A data communication method according to Claim 
30. further comprising a step of dynamically chang- 
ing the priority-level information corresponding to 
the layer of the scalable-encoded data of the image 
signal, stored in the packet, according to a network 
condition, the step being executed by the data 
transmission apparatus. 

38. A data communication method according to Claim 



30, 

wherein the encoding step executed in the da- 
ta transmission apparatus comprises a step of ap- 
plying the scalable-encoding processing by using 
the wavelet transform to the image signal, and of 
generating encoded data having different spatial 
resolutions generated by the wavelet transform, 
and 

the packet generation step comprises a step 
of specifying priority-level information correspond- 
ing to layers of the spatial resolutions of the encod- 
ed data having the different spatial resolutions, as 
additional information of the generated packet. 

39. A data communication method according to Claim 
30, 

wherein the encoding step executed in the da- 
ta transmission apparatus comprises a step of gen- 
erating scalable-encoded data by progressive en- 
coding processing, and 

the packet generation step comprises a step 
of specifying priority-level information correspond- 
ing to a layer specified for a progressive order of the 
scalable-encoded data generated by applying the 
progressive encoding processing to the image sig- 
nal and stored in the generated packet, as addition- 
al information of the generated packet. 

40. A data communication method according to Claim 
30 ; 

wherein the encoding step executed in the da 
ta transmission apparatus comprises a step of gen- 
erating scalable-encoded data by progressive en- 
coding processing in which different spatial resolu- 
tions are specified in a progressive order, and 

the packet generation step comprises a step 
of specifying priority-level information correspond- 
ing to a layer specified for each spatial resolution of 
the scalable-encoded data of the image signal, 
stored in the generated packet, as additional infor- 
mation of the generated packet. 

41. A data communication method according to Claim 
30, 

wherein the encoding step executed in the da- 
ta transmission apparatus comprises a step of gen- 
erating scalable-encoded data by progressive en- 
coding processing in which different SNRs are 
specified in a progressive order, and 

the packet generation step comprises a step 
of specifying priority-level information correspond- 
ing to a layer specified for each SN R of the scalable- 
encoded data of the image signal, stored in the gen- 
erated packet, as additional information of the gen- 
erated packet. 

42. A data communication method according to Claim 
30, 
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wherein the encoding step executed in the da- 
ta transmission apparatus comprises a step of gen- 
erating scalable-encoded data by progressive en- 
coding processing in which different color compo- 
nents are specified in a progressive order, and 

the packet generation step comprises a step 
of specifying priority- level information correspond- 
ing to a layer specified for each color component of 
the scalable-encoded data of the image signal 
stored in the generated packet, as additional infor- 
mation of the generated packet. 

43. A data communication method according to Claim 
30, further comprising a step of, in response to a 
packet re-transmission request sent from the data 
receiving apparatus, determining the levels of pri- 
ority assigned to packets for which the re-transmis- 
sion request has been sent; of determining whether 
re-transmission is to be performed according to the 
determined levels of priority, and of re-transmitting 
packets having higher levels of priority earlier, the 
step being executed by the data transmission ap- 
paratus. 

44. A data transmission method for storing image data 
in a packet and fortransmitting the packet, compris- 
ing the steps of: 

an encoding step of applying scalable-encod- 
ing processing by using wavelet transform to an 
image signal; and 

a packet generation step of generating a packet 
which stores scalable-encoded data generated 
in the encoding step and of specifying priority- 
level information corresponding to a layer of the 
scalable-encoded data of the image signal, 
stored in the generated packet, as additional in- 
formation of the generated packet. 

45. A data transmission method according to Claim 44. 
wherein the packet generation step comprises a 
step of specifying the priority-level information cor- 
responding to the layer of the scalable-encoded da- 
ta of the image signal, in an RTP header. 

46. A data transmission method according to Claim 44. 
wherein the packet generation step comprises a 
step of specifying the priority-level information cor- 
responding to the layer of the scalable-encoded da- 
ta of the image signal, in an IP header. 

47. A data transmission method according to Claim 44. 
wherein the packet generation step comprises a 
step of specifying the priority-level information cor- 
responding to the layer of the scalable-encoded da- 
ta of the image signal, in an RTP header and an IP 
header. 



48. A data transmission method according to Claim 44, 
wherein the packet generation step comprises a 
step of specifying the priority-level information cor- 
responding to the layer of the scalable-encoded da- 

5 ta of the image signal, stored in the generated pack- 

et, as additional information of the generated pack- 
et, according to a priority -level setting map in which 
the layer of the scalable-encoded data is associated 
with a level of priority. 

10 

49. A data transmission method according to Claim 44, 

further comprising a step of receiving receiv- 
ing-terminal information from the data receiving ap- 
paratus, the step being executed by the data trans- 
'5 mission apparatus, 

wherein the packet generation step compris- 
es a step of dynamically changing the priority-level 
information corresponding to the layer of the scala- 
ble-encoded data of the image signal, stored in the 
20 packet, according to the receiving-terminal informa- 
tion. 

50. A data transmission method according to Claim 44, 
further comprising a step of dynamically changing 

25 the priority-level information corresponding to the 
layer of the scalable-encoded data of the image sig- 
nal, stored in the packet, according to a network 
condition, the step being executed by the data 
transmission apparatus. 

30 

51 . A data transmission method according to Claim 44, 

wherein the encoding step comprises a step 
of applying the scalable-encoding processing by 
using the wavelet transform to the image signal, and 
35 of generating encoded data having different spatial 

resolutions generated by the wavelet transform, 
and 

the packet generation step comprises a step 
of specifying priority-level information correspond- 
40 jng to layers of the spatial resolutions of the encod- 

ed data having the different spatial resolutions, as 
additional information of the generated packet. 

52. A data transmission method according to Claim 44, 
45 wherein the encoding step comprises a step 

of generating scalable-encoded data by progres- 
sive encoding processing, and 

the packet generation step comprises a step 
of specifying priority-level information correspond- 
50 ing to a layer specified for a progressive order of the 
scalable-encoded data generated by applying the 
progressive encoding processing to the image sig- 
nal and stored in the generated packet, as addition- 
al information of the generated packet. 

55 

53. A data transmission method according to Claim 44, 

wherein the encoding step comprises a step 
of generating scalable-encoded data by progres- 
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sive encoding processing in which different spatial 
resolutions are specified in a progressive order, and 
the packet generation step comprises a step 
of specifying priority- lev el information correspond- 
ing to a layer specified for each spatial resolution of 
the scalable-encoded data of the image signal, 
stored in the generated packet, as additional infor- 
mation of the generated packet. 

54. A data transmission method according to Claim 44 : 

wherein the encoding step comprises a step 
of generating scalable-encoded data by progres- 
sive encoding processing in which different SNRs 
are specified in a progressive order and 

the packet generation step comprises a step 
of specifying priority-level information correspond- 
ing to a layer specified for each SNR of the scalable- 
encoded data of the image signal, stored in the gen- 
erated packet, as additional information of the gen- 
erated packet. 

55. A data transmission method according to Claim 44. 

wherein the encoding step comprises a step 
of generating scalable-encoded data by progres- 
sive encoding processing in which different color 
components are specified in a progressive order 
and 

the packet generation step comprises a step 
of specifying priority-level information correspond- 
ing to a layer specified for each color component of 
the scalable-encoded data of the image signal, 
stored in the generated packet, as additional infor- 
mation of the generated packet. 
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ing apparatus; and 

a decoding step of decoding data stored in 
packets selected in the packet-priority-level-in- 
formation determination step. 

58. A received-data processing method according to 
Claim 57, 

wherein the packet-priorfty-level-information 
determination step comprises: 

an IP-packet filtering step of determining a level 
of priority assigned to the IP header of each re- 
ceived packet; and 

an RTP-packet-header scanning step of deter- 
mining a level of priority assigned to the RTP 
header of the received packet. 

59. A computer program for executing processing for 
storing image data in a packet and for sending the 
packet, comprising the steps of: 

an encoding step of applying scalable-encod- 
ing processing by using wavelet transform to an 
image signal; and 

a packet generation step of generating a packet 
which stores scalable-encoded data generated 
in the encoding step and of specifying priority- 
level information corresponding to a layer of the 
scalable-encoded data of the image signal, 
stored in the generated packet, as additional in- 
formation of the generated packet. 



56. A data transmission method according to Claim 44 : 35 
further comprising a step of, in response to a packet 
re-transmission request sent from the data receiv- 
ing apparatus, determining the levels of priority as- 
signed to packets for which the re-transmission re- 
quest has been sent; of determining whether re- 40 
transmission is to be performed according to the de- 
termined levels of priority, and of re-transmitting 
packets having higher levels of priority earlier, the 
step being executed by the data transmission ap- 
paratus. 45 



57. A received-data processing method for receiving 
packets which store encoded data and for execut- 
ing data processing, comprising the steps of: 

50 

a receiving step of receiving packets which 
store data scalable-encoded by wavelet trans- 
form; 

a packet-priority-level-information determina- 
tion step of determining whether the packets 55 
have the information of a level of priority equal 
to or larger than a value specified according to 
the processing performance of the data receiv- 
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